Skip to content
Snippets Groups Projects
Commit 5bf3ed18 authored by Paul McCarthy's avatar Paul McCarthy :mountain_bicyclist:
Browse files

Merge branch 'master' into 'master'

Master

See merge request fsl/pytreat-2018-practicals!45
parents d2abfdcc ef633e9d
No related branches found
No related tags found
No related merge requests found
Showing
with 1682 additions and 0 deletions
%% Cell type:markdown id: tags:
# Convolutional Neural Network Example
Build a convolutional neural network with TensorFlow.
This example is using TensorFlow layers API, see 'convolutional_network_raw' example
for a raw TensorFlow implementation with variables.
- Author: Aymeric Damien
- Project: https://github.com/aymericdamien/TensorFlow-Examples/
%% Cell type:markdown id: tags:
## MNIST Dataset Overview
This example is using MNIST handwritten digits. The dataset contains 60,000 examples for training and 10,000 examples for testing. The digits have been size-normalized and centered in a fixed-size image (28x28 pixels) with values from 0 to 1. For simplicity, each image has been flattened and converted to a 1-D numpy array of 784 features (28*28).
![MNIST Dataset](http://neuralnetworksanddeeplearning.com/images/mnist_100_digits.png)
More info: http://yann.lecun.com/exdb/mnist/
## CNN Overview
![CNN](http://personal.ie.cuhk.edu.hk/~ccloy/project_target_code/images/fig3.png)
%% Cell type:code id: tags:
``` python
from __future__ import division, print_function, absolute_import
# Import MNIST data
from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets("/tmp/data/", one_hot=False)
import tensorflow as tf
import matplotlib.pyplot as plt
import numpy as np
```
%% Output
Extracting /tmp/data/train-images-idx3-ubyte.gz
Extracting /tmp/data/train-labels-idx1-ubyte.gz
Extracting /tmp/data/t10k-images-idx3-ubyte.gz
Extracting /tmp/data/t10k-labels-idx1-ubyte.gz
%% Cell type:code id: tags:
``` python
# Training Parameters
learning_rate = 0.001
num_steps = 2000
batch_size = 128
# Network Parameters
num_input = 784 # MNIST data input (img shape: 28*28)
num_classes = 10 # MNIST total classes (0-9 digits)
dropout = 0.25 # Dropout, probability to drop a unit
```
%% Cell type:code id: tags:
``` python
# Create the neural network
def conv_net(x_dict, n_classes, dropout, reuse, is_training):
# Define a scope for reusing the variables
with tf.variable_scope('ConvNet', reuse=reuse):
# TF Estimator input is a dict, in case of multiple inputs
x = x_dict['images']
# MNIST data input is a 1-D vector of 784 features (28*28 pixels)
# Reshape to match picture format [Height x Width x Channel]
# Tensor input become 4-D: [Batch Size, Height, Width, Channel]
x = tf.reshape(x, shape=[-1, 28, 28, 1])
# Convolution Layer with 32 filters and a kernel size of 5
conv1 = tf.layers.conv2d(x, 32, 5, activation=tf.nn.relu)
# Max Pooling (down-sampling) with strides of 2 and kernel size of 2
conv1 = tf.layers.max_pooling2d(conv1, 2, 2)
# Convolution Layer with 64 filters and a kernel size of 3
conv2 = tf.layers.conv2d(conv1, 64, 3, activation=tf.nn.relu)
# Max Pooling (down-sampling) with strides of 2 and kernel size of 2
conv2 = tf.layers.max_pooling2d(conv2, 2, 2)
# Flatten the data to a 1-D vector for the fully connected layer
fc1 = tf.contrib.layers.flatten(conv2)
# Fully connected layer (in tf contrib folder for now)
fc1 = tf.layers.dense(fc1, 1024)
# Apply Dropout (if is_training is False, dropout is not applied)
fc1 = tf.layers.dropout(fc1, rate=dropout, training=is_training)
# Output layer, class prediction
out = tf.layers.dense(fc1, n_classes)
return out
```
%% Cell type:code id: tags:
``` python
# Define the model function (following TF Estimator Template)
def model_fn(features, labels, mode):
# Build the neural network
# Because Dropout have different behavior at training and prediction time, we
# need to create 2 distinct computation graphs that still share the same weights.
logits_train = conv_net(features, num_classes, dropout, reuse=False, is_training=True)
logits_test = conv_net(features, num_classes, dropout, reuse=True, is_training=False)
# Predictions
pred_classes = tf.argmax(logits_test, axis=1)
pred_probas = tf.nn.softmax(logits_test)
# If prediction mode, early return
if mode == tf.estimator.ModeKeys.PREDICT:
return tf.estimator.EstimatorSpec(mode, predictions=pred_classes)
# Define loss and optimizer
loss_op = tf.reduce_mean(tf.nn.sparse_softmax_cross_entropy_with_logits(
logits=logits_train, labels=tf.cast(labels, dtype=tf.int32)))
optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate)
train_op = optimizer.minimize(loss_op, global_step=tf.train.get_global_step())
# Evaluate the accuracy of the model
acc_op = tf.metrics.accuracy(labels=labels, predictions=pred_classes)
# TF Estimators requires to return a EstimatorSpec, that specify
# the different ops for training, evaluating, ...
estim_specs = tf.estimator.EstimatorSpec(
mode=mode,
predictions=pred_classes,
loss=loss_op,
train_op=train_op,
eval_metric_ops={'accuracy': acc_op})
return estim_specs
```
%% Cell type:code id: tags:
``` python
# Build the Estimator (i.e. class for training and evaluating network)
model = tf.estimator.Estimator(model_fn)
```
%% Output
WARNING:tensorflow:Using temporary folder as model directory: /var/folders/c3/rhg93t4n0cjbgvkb9tp99_h40000gq/T/tmpqhkrjnut
INFO:tensorflow:Using default config.
INFO:tensorflow:Using config: {}
%% Cell type:code id: tags:
``` python
# Define the input function for training
input_fn = tf.estimator.inputs.numpy_input_fn(
x={'images': mnist.train.images}, y=mnist.train.labels,
batch_size=batch_size, num_epochs=None, shuffle=True)
# Train the Model
model.train(input_fn, steps=num_steps)
```
%% Output
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Saving checkpoints for 1 into /var/folders/c3/rhg93t4n0cjbgvkb9tp99_h40000gq/T/tmpqhkrjnut/model.ckpt.
INFO:tensorflow:step = 1, loss = 2.32753
INFO:tensorflow:global_step/sec: 12.1422
INFO:tensorflow:step = 101, loss = 0.127041 (8.237 sec)
INFO:tensorflow:global_step/sec: 11.4857
INFO:tensorflow:step = 201, loss = 0.0393114 (8.707 sec)
INFO:tensorflow:global_step/sec: 8.48714
INFO:tensorflow:step = 301, loss = 0.0843611 (11.783 sec)
INFO:tensorflow:global_step/sec: 7.15251
INFO:tensorflow:step = 401, loss = 0.0262136 (13.981 sec)
INFO:tensorflow:global_step/sec: 6.4515
INFO:tensorflow:step = 501, loss = 0.0497303 (15.500 sec)
INFO:tensorflow:global_step/sec: 6.70897
INFO:tensorflow:step = 601, loss = 0.0142919 (14.905 sec)
INFO:tensorflow:global_step/sec: 6.6801
INFO:tensorflow:step = 701, loss = 0.042797 (14.970 sec)
INFO:tensorflow:global_step/sec: 6.33292
INFO:tensorflow:step = 801, loss = 0.0159051 (15.790 sec)
INFO:tensorflow:global_step/sec: 6.51359
INFO:tensorflow:step = 901, loss = 0.0560886 (15.352 sec)
INFO:tensorflow:global_step/sec: 7.00529
INFO:tensorflow:step = 1001, loss = 0.0389398 (14.275 sec)
INFO:tensorflow:global_step/sec: 11.4372
INFO:tensorflow:step = 1101, loss = 0.000826523 (8.743 sec)
INFO:tensorflow:global_step/sec: 11.8734
INFO:tensorflow:step = 1201, loss = 0.0288895 (8.422 sec)
INFO:tensorflow:global_step/sec: 11.4033
INFO:tensorflow:step = 1301, loss = 0.0109369 (8.769 sec)
INFO:tensorflow:global_step/sec: 12.1424
INFO:tensorflow:step = 1401, loss = 0.016869 (8.236 sec)
INFO:tensorflow:global_step/sec: 12.2184
INFO:tensorflow:step = 1501, loss = 0.025425 (8.184 sec)
INFO:tensorflow:global_step/sec: 12.9909
INFO:tensorflow:step = 1601, loss = 0.00609953 (7.698 sec)
INFO:tensorflow:global_step/sec: 12.3233
INFO:tensorflow:step = 1701, loss = 0.0149776 (8.114 sec)
INFO:tensorflow:global_step/sec: 12.452
INFO:tensorflow:step = 1801, loss = 0.0356573 (8.031 sec)
INFO:tensorflow:global_step/sec: 12.6611
INFO:tensorflow:step = 1901, loss = 0.0126303 (7.898 sec)
INFO:tensorflow:Saving checkpoints for 2000 into /var/folders/c3/rhg93t4n0cjbgvkb9tp99_h40000gq/T/tmpqhkrjnut/model.ckpt.
INFO:tensorflow:Loss for final step: 0.0101263.
<tensorflow.python.estimator.estimator.Estimator at 0x124517438>
%% Cell type:code id: tags:
``` python
# Evaluate the Model
# Define the input function for evaluating
input_fn = tf.estimator.inputs.numpy_input_fn(
x={'images': mnist.test.images}, y=mnist.test.labels,
batch_size=batch_size, shuffle=False)
# Use the Estimator 'evaluate' method
model.evaluate(input_fn)
```
%% Output
INFO:tensorflow:Starting evaluation at 2018-02-22-18:15:46
INFO:tensorflow:Restoring parameters from /var/folders/c3/rhg93t4n0cjbgvkb9tp99_h40000gq/T/tmpqhkrjnut/model.ckpt-2000
INFO:tensorflow:Finished evaluation at 2018-02-22-18:15:48
INFO:tensorflow:Saving dict for global step 2000: accuracy = 0.9878, global_step = 2000, loss = 0.0447589
WARNING:tensorflow:Skipping summary for global_step, must be a float or np.float32.
{'accuracy': 0.9878, 'global_step': 2000, 'loss': 0.044758931}
%% Cell type:code id: tags:
``` python
# Predict single images
n_images = 4
# Get images from test set
test_images = mnist.test.images[:n_images]
# Prepare the input data
input_fn = tf.estimator.inputs.numpy_input_fn(
x={'images': test_images}, shuffle=False)
# Use the model to predict the images class
preds = list(model.predict(input_fn))
# Display
for i in range(n_images):
plt.imshow(np.reshape(test_images[i], [28, 28]), cmap='gray')
plt.show()
print("Model prediction:", preds[i])
```
%% Output
INFO:tensorflow:Restoring parameters from /var/folders/c3/rhg93t4n0cjbgvkb9tp99_h40000gq/T/tmpqhkrjnut/model.ckpt-2000
Model prediction: 7
Model prediction: 2
Model prediction: 1
Model prediction: 0
%% Cell type:code id: tags:
``` python
```
7
2
1
0
4
1
4
9
5
9
0
6
9
0
1
5
9
7
3
4
9
6
6
5
4
0
7
4
0
1
3
1
3
4
7
2
7
1
2
1
1
7
4
2
3
5
1
2
4
4
6
3
5
5
6
0
4
1
9
5
7
8
9
3
7
4
6
4
3
0
7
0
2
9
1
7
3
2
9
7
7
6
2
7
8
4
7
3
6
1
3
6
9
3
1
4
1
7
6
9
6
0
5
4
9
9
2
1
9
4
8
7
3
9
7
4
4
4
9
2
5
4
7
6
7
9
0
5
8
5
6
6
5
7
8
1
0
1
6
4
6
7
3
1
7
1
8
2
0
2
9
9
5
5
1
5
6
0
3
4
4
6
5
4
6
5
4
5
1
4
4
7
2
3
2
7
1
8
1
8
1
8
5
0
8
9
2
5
0
1
1
1
0
9
0
3
1
6
4
2
3
6
1
1
1
3
9
5
2
9
4
5
9
3
9
0
3
6
5
5
7
2
2
7
1
2
8
4
1
7
3
3
8
8
7
9
2
2
4
1
5
9
8
7
2
3
0
4
4
2
4
1
9
5
7
7
2
8
2
6
8
5
7
7
9
1
8
1
8
0
3
0
1
9
9
4
1
8
2
1
2
9
7
5
9
2
6
4
1
5
8
2
9
2
0
4
0
0
2
8
4
7
1
2
4
0
2
7
4
3
3
0
0
3
1
9
6
5
2
5
9
2
9
3
0
4
2
0
7
1
1
2
1
5
3
3
9
7
8
6
5
6
1
3
8
1
0
5
1
3
1
5
5
6
1
8
5
1
7
9
4
6
2
2
5
0
6
5
6
3
7
2
0
8
8
5
4
1
1
4
0
3
3
7
6
1
6
2
1
9
2
8
6
1
9
5
2
5
4
4
2
8
3
8
2
4
5
0
3
1
7
7
5
7
9
7
1
9
2
1
4
2
9
2
0
4
9
1
4
8
1
8
4
5
9
8
8
3
7
6
0
0
3
0
2
6
6
4
9
3
3
3
2
3
9
1
2
6
8
0
5
6
6
6
3
8
8
2
7
5
8
9
6
1
8
4
1
2
5
9
1
9
7
5
4
0
8
9
9
1
0
5
2
3
7
8
9
4
0
6
3
9
5
2
1
3
1
3
6
5
7
4
2
2
6
3
2
6
5
4
8
9
7
1
3
0
3
8
3
1
9
3
4
4
6
4
2
1
8
2
5
4
8
8
4
0
0
2
3
2
7
7
0
8
7
4
4
7
9
6
9
0
9
8
0
4
6
0
6
3
5
4
8
3
3
9
3
3
3
7
8
0
8
2
1
7
0
6
5
4
3
8
0
9
6
3
8
0
9
9
6
8
6
8
5
7
8
6
0
2
4
0
2
2
3
1
9
7
5
1
0
8
4
6
2
6
7
9
3
2
9
8
2
2
9
2
7
3
5
9
1
8
0
2
0
5
2
1
3
7
6
7
1
2
5
8
0
3
7
2
4
0
9
1
8
6
7
7
4
3
4
9
1
9
5
1
7
3
9
7
6
9
1
3
7
8
3
3
6
7
2
8
5
8
5
1
1
4
4
3
1
0
7
7
0
7
9
4
4
8
5
5
4
0
8
2
1
0
8
4
5
0
4
0
6
1
7
3
2
6
7
2
6
9
3
1
4
6
2
5
4
2
0
6
2
1
7
3
4
1
0
5
4
3
1
1
7
4
9
9
4
8
4
0
2
4
5
1
1
6
4
7
1
9
4
2
4
1
5
5
3
8
3
1
4
5
6
8
9
4
1
5
3
8
0
3
2
5
1
2
8
3
4
4
0
8
8
3
3
1
7
3
5
9
6
3
2
6
1
3
6
0
7
2
1
7
1
4
2
4
2
1
7
9
6
1
1
2
4
8
1
7
7
4
8
0
7
3
1
3
1
0
7
7
0
3
5
5
2
7
6
6
9
2
8
3
5
2
2
5
6
0
8
2
9
2
8
8
8
8
7
4
9
3
0
6
6
3
2
1
3
2
2
9
3
0
0
5
7
8
1
4
4
6
0
2
9
1
4
7
4
7
3
9
8
8
4
7
1
2
1
2
2
3
2
3
2
3
9
1
7
4
0
3
5
5
8
6
3
2
6
7
6
6
3
2
7
8
1
1
7
5
6
4
9
5
1
3
3
4
7
8
9
1
1
6
9
1
4
4
5
4
0
6
2
2
3
1
5
1
2
0
3
8
1
2
6
7
1
6
2
3
9
0
1
2
2
0
8
9
9
0
2
5
1
9
7
8
1
0
4
1
7
9
6
4
2
6
8
1
3
7
5
4
talks/tensorflow/demos/sprite_1024.png

169 KiB

%% Cell type:code id: tags:
``` python
import os
import os.path
import shutil
import tensorflow as tf
LOGDIR = "/tmp/TensorBoard_demo/"
LABELS = os.path.join(os.getcwd(), "labels_1024.tsv")
SPRITES = os.path.join(os.getcwd(), "sprite_1024.png")
### MNIST EMBEDDINGS ###
mnist = tf.contrib.learn.datasets.mnist.read_data_sets(train_dir=LOGDIR + "data", one_hot=True)
### Get a sprite and labels file for the embedding projector ###
if not (os.path.isfile(LABELS) and os.path.isfile(SPRITES)):
print("Necessary data files were not found: LABELS and SPRITES")
exit(1)
```
%% Output
Extracting /tmp/TensorBoard_demo/data/train-images-idx3-ubyte.gz
Extracting /tmp/TensorBoard_demo/data/train-labels-idx1-ubyte.gz
Extracting /tmp/TensorBoard_demo/data/t10k-images-idx3-ubyte.gz
Extracting /tmp/TensorBoard_demo/data/t10k-labels-idx1-ubyte.gz
%% Cell type:code id: tags:
``` python
def conv_layer(input, size_in, size_out, name="conv"):
with tf.name_scope(name):
w = tf.Variable(tf.truncated_normal([5, 5, size_in, size_out], stddev=0.1), name="W")
b = tf.Variable(tf.constant(0.1, shape=[size_out]), name="B")
conv = tf.nn.conv2d(input, w, strides=[1, 1, 1, 1], padding="SAME")
act = tf.nn.relu(conv + b)
tf.summary.histogram("weights", w)
tf.summary.histogram("biases", b)
tf.summary.histogram("activations", act)
return tf.nn.max_pool(act, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding="SAME")
def fc_layer(input, size_in, size_out, name="fc"):
with tf.name_scope(name):
w = tf.Variable(tf.truncated_normal([size_in, size_out], stddev=0.1), name="W")
b = tf.Variable(tf.constant(0.1, shape=[size_out]), name="B")
act = tf.matmul(input, w) + b
tf.summary.histogram("weights", w)
tf.summary.histogram("biases", b)
tf.summary.histogram("activations", act)
return act
```
%% Cell type:code id: tags:
``` python
def mnist_model(learning_rate):
tf.reset_default_graph()
sess = tf.Session()
# Setup placeholders, and reshape the data
x = tf.placeholder(tf.float32, shape=[None, 784], name="x")
x_image = tf.reshape(x, [-1, 28, 28, 1])
tf.summary.image('input', x_image, 3)
y = tf.placeholder(tf.float32, shape=[None, 10], name="labels")
conv_out = conv_layer(x_image, 1, 16, "conv")
flattened = tf.reshape(conv_out, [-1, 7 * 7 * 64])
embedding_input = flattened
embedding_size = 7*7*64
logits = fc_layer(flattened, 7*7*64, 10, "fc")
with tf.name_scope("xent"):
xent = tf.reduce_mean(
tf.nn.softmax_cross_entropy_with_logits(
logits=logits, labels=y), name="xent")
tf.summary.scalar("xent", xent)
with tf.name_scope("train"):
train_step = tf.train.AdamOptimizer(learning_rate).minimize(xent)
with tf.name_scope("accuracy"):
correct_prediction = tf.equal(tf.argmax(logits, 1), tf.argmax(y, 1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
tf.summary.scalar("accuracy", accuracy)
summ = tf.summary.merge_all()
embedding = tf.Variable(tf.zeros([1024, embedding_size]), name="test_embedding")
assignment = embedding.assign(embedding_input)
saver = tf.train.Saver()
sess.run(tf.global_variables_initializer())
writer = tf.summary.FileWriter(LOGDIR)
writer.add_graph(sess.graph)
config = tf.contrib.tensorboard.plugins.projector.ProjectorConfig()
embedding_config = config.embeddings.add()
embedding_config.tensor_name = embedding.name
embedding_config.sprite.image_path = SPRITES
embedding_config.metadata_path = LABELS
# Specify the width and height of a single thumbnail.
embedding_config.sprite.single_image_dim.extend([28, 28])
tf.contrib.tensorboard.plugins.projector.visualize_embeddings(writer, config)
for i in range(1,2001):
batch = mnist.train.next_batch(100)
if i % 5 == 0:
[train_accuracy, s] = sess.run([accuracy, summ], feed_dict={x: batch[0], y: batch[1]})
writer.add_summary(s, i)
if i % 500 == 0:
sess.run(assignment, feed_dict={x: mnist.test.images[:1024], y: mnist.test.labels[:1024]})
saver.save(sess, os.path.join(LOGDIR, "model.ckpt"), i)
sess.run(train_step, feed_dict={x: batch[0], y: batch[1]})
```
%% Cell type:code id: tags:
``` python
learning_rate = 1E-3
mnist_model(learning_rate)
print('Done training!')
print('Run `tensorboard --logdir=%s` to see the results.' % LOGDIR)
print('Running on mac? If you want to get rid of the dialogue asking to give '
'network permissions to TensorBoard, you can provide this flag: '
'--host=localhost')
```
%% Output
Done training!
Run `tensorboard --logdir=/tmp/TensorBoard_demo/` to see the results.
Running on mac? If you want to get rid of the dialogue asking to give network permissions to TensorBoard, you can provide this flag: --host=localhost
%% Cell type:code id: tags:
``` python
!tensorboard --logdir=/tmp/TensorBoard_demo/
```
%% Output
Starting TensorBoard b'47' at http://0.0.0.0:6006
(Press CTRL+C to quit)
WARNING:tensorflow:Found more than one graph event per run, or there was a metagraph containing a graph_def, as well as one or more graph events. Overwriting the graph with the newest event.
WARNING:tensorflow:Found more than one graph event per run, or there was a metagraph containing a graph_def, as well as one or more graph events. Overwriting the graph with the newest event.
WARNING:tensorflow:Found more than one graph event per run, or there was a metagraph containing a graph_def, as well as one or more graph events. Overwriting the graph with the newest event.
WARNING:tensorflow:path ../external/data/plugin/text/runs not found, sending 404
WARNING:tensorflow:path ../external/data/plugin/text/runs not found, sending 404
WARNING:tensorflow:path ../external/data/plugin/text/runs not found, sending 404
WARNING:tensorflow:path ../external/data/plugin/text/runs not found, sending 404
^C
%% Cell type:code id: tags:
``` python
```
talks/tensorflow/img/bigger_boat.jpg

77.3 KiB

talks/tensorflow/img/cg/ad.png

156 KiB

talks/tensorflow/img/cg/cg1.png

113 KiB

talks/tensorflow/img/copyright_infringement.png

419 KiB

talks/tensorflow/img/fchollet_popularity_2017.jpg

91.9 KiB

talks/tensorflow/img/how_big_is_your_data.png

16.1 KiB

talks/tensorflow/img/model_zoo.png

62.7 KiB

talks/tensorflow/img/multiple_devices.png

151 KiB

talks/tensorflow/img/so_hot_right_now.jpg

78.7 KiB

talks/tensorflow/img/what_leo_needs.png

280 KiB

0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment