We developed a parallel neural network framework running well on iOS devices despite of the limited memory and computing resources. Our framework features low memory footprint and high parallelism. By extensively using CPU SIMD operations, GPU acceleration, on-demand output, on-the-fly network decompression, sparse matrix operations and many other techniques, one can evaluate networks as deep as 60+ layers or as large as AlexNet1 with ease.