{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"name":"TransporterNeurIPS2019","provenance":[],"collapsed_sections":["cjRprVl171mo"],"last_runtime":{"build_target":"//learning/deepmind/dm_python:dm_notebook3","kind":"private"},"toc_visible":true},"kernelspec":{"name":"python3","display_name":"Python 3"}},"cells":[{"cell_type":"markdown","metadata":{"id":"cjRprVl171mo","colab_type":"text"},"source":["##### Copyright 2019 Google LLC.\n","\n","Licensed under the Apache License, Version 2.0 (the \"License\");"]},{"cell_type":"code","metadata":{"id":"0dS1sgk6Pc20","colab_type":"code","colab":{}},"source":["# Licensed under the Apache License, Version 2.0 (the \"License\");\n","# you may not use this file except in compliance with the License.\n","# You may obtain a copy of the License at\n","#\n","# https://www.apache.org/licenses/LICENSE-2.0\n","#\n","# Unless required by applicable law or agreed to in writing, software\n","# distributed under the License is distributed on an \"AS IS\" BASIS,\n","# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n","# See the License for the specific language governing permissions and\n","# limitations under the License."],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"9EbAW6qLPu0b","colab_type":"text"},"source":["# Transporter\n","\n","## Introduction\n","\n","This notebook contains an example describing how to use the `Transporter` architecture as described in [Unsupervised Learning of Object Keypoints for Perception and Control](https://arxiv.org/abs/1906.11883).\n","\n","In this notebook, we show how to create a model, and how to use it to perform inference on single images. The model is loaded from a pre-trained Transporter model checkpoint. The data consists of a full episode of Pong obtained from a random agent.\n","\n","## Contact\n","For questions or bug reports, please create an issue in the [GitHub repository](https://github.com/deepmind/deepmind-research/transporter)."]},{"cell_type":"markdown","metadata":{"id":"Lrq82SrnPbzd","colab_type":"text"},"source":["## Load files into colab\n","To run this colab you will need to upload the `transporter.py` file. \n","\n","You can do this by dragging and dropping the files into the `Files` tab in the left pane."]},{"cell_type":"code","metadata":{"id":"f1xXOVQDdYwu","colab_type":"code","cellView":"form","colab":{}},"source":["#@title Load Pong frames and checkpoint from GCS\n","!gsutil cp gs://transporter-data/pong_episode.npy ./pong_episode.npy\n","!gsutil cp gs://transporter-data/transporter_ckpt* ./"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"Be3deVlENnYn","colab_type":"code","cellView":"form","colab":{}},"source":["#@title Imports\n","from matplotlib import animation, cm\n","from matplotlib import pyplot as plt\n","import numpy as np\n","import PIL\n","from PIL import Image, ImageDraw\n","import tensorflow as tf\n","\n","from IPython.display import HTML\n","from base64 import b64encode\n","\n","import transporter"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"dupTYWxlBqDo","colab_type":"code","cellView":"both","colab":{}},"source":["#@title Plotting functions\n","def plot_keypoints_on_image(im, kpts, r=5, alpha=1.0, image_size=(128, 128)):\n"," im = (im.copy() * alpha).astype(np.uint8)\n"," scaled_kpts = kpts.copy()\n"," scaled_kpts = (scaled_kpts + 1.0) * 0.5 * np.array(list(image_size))[None,:]\n"," scaled_kpts = scaled_kpts.astype(np.int)\n"," temp = scaled_kpts.copy()\n"," if not isinstance(im, Image.Image):\n"," im = Image.fromarray(im)\n"," \n"," im = im.resize(image_size, PIL.Image.BILINEAR)\n"," n_kpts = scaled_kpts.shape[0]\n"," colors = [tuple(c) for c in (cm.rainbow(np.linspace(0,1,n_kpts)) * 255).astype(np.int32)]\n"," draw = ImageDraw.Draw(im)\n"," for i in range(len(scaled_kpts)):\n"," y,x = scaled_kpts[i].astype(np.int32)\n"," draw.ellipse((x-r, y-r, x+r, y+r), fill=colors[i])\n"," return np.array(im), temp\n","\n","def save_video(frames, fname):\n"," fig = plt.figure(figsize=(10, 10))\n"," video = []\n"," for frame in frames:\n"," im = plt.imshow(frame, animated=True, interpolation='nearest')\n"," video.append([im])\n"," ani = animation.ArtistAnimation(\n"," fig, video, interval=50, blit=True, repeat_delay=1000)\n"," ani.save(fname)"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"p_ouwl7XITCO","colab_type":"code","cellView":"both","colab":{}},"source":["#@title Model params\n","ckpt_path = 'transporter_ckpt'\n","\n","norm_type = 'instance'\n","convnet_encoder_kwargs=dict(\n"," filters=(16, 16, 32, 32),\n"," strides=(1, 1, 2, 1),\n"," kernel_sizes=(7, 3, 3, 3),\n",")\n","decoder_num_filters=128\n","\n","num_keypoints=5\n","gauss_std=0.1\n","norm_type='instance'\n","\n","image_shape = (128, 128)"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"c5jN2LNZRVlB","colab_type":"code","cellView":"both","colab":{"base_uri":"https://localhost:8080/","height":309},"outputId":"99a9737e-6416-4c1b-e89f-2f6448c08026","executionInfo":{"status":"ok","timestamp":1571160570808,"user_tz":-60,"elapsed":5952,"user":{"displayName":"Sebastian Borgeaud","photoUrl":"","userId":"12373152407193217991"}}},"source":["#@title Build the model\n","tf.reset_default_graph()\n","encoder = transporter.Encoder(\n"," name='encoder',\n"," norm_type=norm_type,\n"," **convnet_encoder_kwargs)\n","keypoint_encoder = transporter.Encoder(\n"," name='keypoint_encoder',\n"," norm_type=norm_type,\n"," **convnet_encoder_kwargs)\n","\n","decoder = transporter.Decoder(\n"," initial_filters=decoder_num_filters,\n"," output_size=image_shape,\n"," norm_type=norm_type)\n","\n","keypointer = transporter.KeyPointer(\n"," num_keypoints=num_keypoints,\n"," gauss_std=gauss_std,\n"," keypoint_encoder=keypoint_encoder)\n","\n","model = transporter.Transporter(\n"," encoder=encoder,\n"," decoder=decoder,\n"," keypointer=keypointer)\n","\n","image_a_pl = tf.placeholder(\n"," shape=(1, image_shape[0], image_shape[1], 3), \n"," dtype=tf.float32)\n","\n","image_b_pl = tf.placeholder(\n"," shape=(1, image_shape[0], image_shape[1], 3), \n"," dtype=tf.float32)\n","\n","outputs_tf = model(image_a_pl, image_b_pl, is_training=False)"],"execution_count":5,"outputs":[{"output_type":"stream","text":["WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/sonnet/python/modules/base.py:177: The name tf.make_template is deprecated. Please use tf.compat.v1.make_template instead.\n","\n","WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/sonnet/python/modules/base.py:278: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.\n","\n","WARNING:tensorflow:From /content/transporter.py:196: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.\n","\n","WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/sonnet/python/modules/conv.py:299: calling TruncatedNormal.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.\n","Instructions for updating:\n","Call initializer instance with the dtype argument instead of passing it to the constructor\n","WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/sonnet/python/modules/conv.py:665: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead.\n","\n","WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/sonnet/python/modules/conv.py:304: calling Zeros.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.\n","Instructions for updating:\n","Call initializer instance with the dtype argument instead of passing it to the constructor\n","WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/sonnet/python/modules/conv.py:134: The name tf.Dimension is deprecated. Please use tf.compat.v1.Dimension instead.\n","\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"IKUelZxwRbHv","colab_type":"code","cellView":"both","outputId":"571af991-2421-411a-aef1-2005e3175439","executionInfo":{"status":"ok","timestamp":1571160705702,"user_tz":-60,"elapsed":140572,"user":{"displayName":"Sebastian Borgeaud","photoUrl":"","userId":"12373152407193217991"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"source":["#@title Run the model and extract the keypoints\n","input_pixels = np.load('pong_episode.npy')\n","\n","# Add ops to save and restore all the variables.\n","variables_map = {v.name[:-2]: v for v in tf.trainable_variables()}\n","saver = tf.train.Saver(var_list=variables_map)\n","\n","# Later, launch the model, use the saver to restore variables from disk, and\n","# do some work with the model.\n","with tf.Session() as sess:\n"," # Restore variables from disk.\n"," saver.restore(sess, ckpt_path)\n","\n"," im_keypoints = []\n"," reconstructed_frames = []\n"," for im in input_pixels:\n"," output = sess.run(outputs_tf, {\n"," image_a_pl: im[None], # Add extra batch dimension\n"," image_b_pl: im[None], \n"," })\n"," im_keypoints.append(output['keypoints_b']['centers'][0])\n"," reconstructed_frames.append(output['reconstructed_image_b'][0])"],"execution_count":6,"outputs":[{"output_type":"stream","text":["INFO:tensorflow:Restoring parameters from transporter_ckpt\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"BYaZD2BH1nO-","colab_type":"code","cellView":"form","colab":{}},"source":["#@title Create output videos\n","keypoint_frames = []\n","scaled_kpts = []\n","for i in range(len(im_keypoints)):\n"," keypoints = im_keypoints[i]\n"," kpt_frame, temp = plot_keypoints_on_image(input_pixels[i], keypoints, r=3)\n"," keypoint_frames.append(kpt_frame)\n"," scaled_kpts.append(temp)\n","keypoint_frames = np.array(keypoint_frames)\n","\n","reconstructed_frames = np.clip(\n"," np.array(reconstructed_frames), 0, 255).astype(np.uint8)\n","\n","collage = np.concatenate(\n"," [input_pixels, keypoint_frames, reconstructed_frames], axis=2)\n"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"7OkUVcx9GbVo","colab_type":"code","outputId":"9f30c0a6-9490-4c93-a9f1-27025a54e674","executionInfo":{"status":"ok","timestamp":1571160788155,"user_tz":-60,"elapsed":222643,"user":{"displayName":"Sebastian Borgeaud","photoUrl":"","userId":"12373152407193217991"}},"cellView":"form","colab":{"base_uri":"https://localhost:8080/","height":237}},"source":["#@title Save the video to the Colab files\n","\n","save_video(collage, 'collage.mp4')"],"execution_count":8,"outputs":[{"output_type":"display_data","data":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAlkAAADcCAYAAABDGrQFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3X2wJXV95/HP9z4xDAMzA4PswAzy\n6FqIWaQmxOiWRUlUxKxDqiiDu6WsoYrNBl3ZuBUxqS2S3bVK9yFqdmtNTQI6WgQkqAVlGRIyYNyt\nWtAhTuRZR5AwODCIMwMDM3Pn3PvdP7pPn1/36e7T56HP032/pu7c7l8//U6f373ne39Pbe4uAAAA\nDNbMqDMAAAAwjQiyAAAAakCQBQAAUAOCLAAAgBoQZAEAANSAIAsAAKAGtQVZZna5mT1pZrvN7Ma6\nrgMAADCOrI55ssxsVtKPJL1L0h5J35f0QXd/bOAXAwAAGEN11WRdImm3uz/l7ouSbpe0taZrAQAA\njJ25ms57hqRng/U9kn6laOf5WfNVc1Z6wtXzpuNmy/fJPfd89BJN1Y9dWl6Ovi8tJWlmpvm57m9X\nIz7HcnxOVOczpsaq+a6Pm12M7vlMY6nDnsO1/0jnMkA5X3lWajlfiMt5Xolttq9ktzXLeVFJz7bL\nmIrL+VxQzqv+1FDOu9d8T3zGtDRF5fzZl5d+7u6ndtqvriCrIzO7TtJ1krR+1YxuunTtwK8xPzen\nt1/8lmS5qmeff16S9MRTTydp6048Ub/85gu7zkPzHM1zorpDp6/XE1e/revjzrzvEUnS63Y9M+gs\njSXK+WSbxnIe9kIxC9O8lRjvZGbBcZ46JjoiWlkIyvnc7Gzp9S1YSsr502E5X6MtF6bLeXJNzwR5\nQfrjSTl/IdjflQ7TWut5QV9e+jirmuf0HQjuYLzh0Onr9ORvdl/ON8fl/LRdz8jjm245mamSv6r3\n3bJvZ/MEQeIN9xyo9INXV3Phc5I2B+ub4rSEu29z9y3uvmXNQvd/uQMAAIyzumqyvi/pfDM7W1Fw\ndbWkf1nTtUotHjsmSVou6eA/Nzur2Znu483mucMzz8/NacYIGmvhrvnDi/FyK7lx/Ly8h/dvmlDO\np8iUlfN0kSwvM2aWqW5Il+fFxagsLs0t5x7jcs3PzWp2ZrZ1eFjjVFJmjy4eaztkbm5es0EVV7PW\nLfrWrJXLHhXWo2VeiuWkT4jwJee9gKT2yq1oh8x6lXJuqbtpJdevck/b9sn7dZndqc83q5Ygy90b\nZvZRSX8taVbSLe7+aB3XKtNYWtKux5+IVkpu1DmbN2vjhg09n/vYUiNJf/Mb3qCTTjih67yis9mj\nDZ13185o+cixJP2p912s11530qiyNXJVy/nPvr6tcNu6Kz7c8dyU8+GYhnKe20QYSsU7RY1q8XIc\npTWWlrTriagsurWajLKNdWdv3pT/+7wkwFoKynljaSk5YVTOVxccldemlFnNtlGFy5MWaWVfR+HL\nz/9DL/tyZ482dO7dD0mS5o4cS97ndDn3pImw/SRd3sxsnsPDPAwS0zuEzci9vGe19cly929L+nZd\n5wcAABhnI+v4PgzurteOHOm4X6PR6LhP2bmPBccz6qQ+5q7jDrwmSZprVjNr/EadDFuVcn7g21+p\ntH3db15feG7K+XBMUzkvbVHusve3+7JeO3I4PjT/xCZpqZEeRZh3wVQdiJlcrsNxOV9sNJLjlpaW\nFVZ3tJo+SzIf1t55M1fNFcvsN4YKa4syqua/YD9z16oDr0qS5g4fS3bMlnML25s9qCJMvbUerFet\nbnLlVl6F1y6tZa1mqoOs2ZkZnXn6xni5eDTK2jVrhpUlYOCqlPOdw8wQkPDcT7DiUXcW/x+mRMsz\nMzN6/caonM/Mzua2Vpmi3+dWFN155rsUfUC7JYFbW+tUxQm7092FgsCssL1QqT3HRsW4qmjUYTej\n+JJbW+Uep07sxYeUnCs7KrE4uMrZqcc3avJ6UQIAAEyAqa7JmpmZ0etPP11Sd/MHAZOkSjmnJgsj\n02GipXTNR3qnsFJidmZGZ54R1WTNz823tRY1jy+qxfKgE3NqDzO5eU7TUNwZOsh/7qmbrYBe9CrK\njWuroVRceWMV9mltLalZ6jpHJSrcyCr32gpXqp4hbUVGHoePHi3tU9IcygtMsk7lHOPpS7Za+sh7\nc7d9SN33Hx22tkFYJcFV+Vmae3krJad56fDiYjCbu7VNJNn8fW6lk4RaEkGF3XtMllwqFUe1TUfQ\nekU+1mFTd1IzYBSkt0tPYVF2N3zs2kozr60tg56fXILmQgAAgBqsyJqsJ37ylA4eOlS4vWxCR2BS\ndCrnGD9fPVL+K7m5/UOrxrdGy1J1AcGEVuE+mR7UxZNMenimzJbI4z/5iV5Oynn7XkvL+ZNWNTvU\nu3vyJUUVWmGFlYWTfuW1k3l6dVpVfn0Wjsqz3EfgJLt21a6aHkFo2UEVHZqluzLAnu8rMshqLC2l\nhqMD0ygs5+uu+HDpNA7JZKQnnjiMrGFFaQ9S0p+NVvChVnyqcLHRCH+ft38QhrMuWNApq71Zr9nc\nZ5ksBAFDM8WCtbZLTmm4VTqbQfi0wh6mPahyyyzb+JgfwJfPG1Igddqy0aDNsaPVIzmaCwEAAGqw\nImuyzt28WYuN4s7tLx04KEn62b59hfvMzs7qgnPPlSQte6tz8erjjx9QLpG1tDCnZ971ZkmSLbXu\n+ZH1PN4lT1s5v+E/p7ZTzlE3z/SQNs9p0wl3ytRMWEETX3M2LUk678zNOnrsWLI9W0PVKucvJk2C\nzXM0zc3N6U3nnSNJWlpeTmpDVh+/Kuj47pmO79m6m/ZzT3PHk+Z7GTeyJkupSiGTvKBNcGlhTj99\n9y9JSk9A2vb7PGmuTS97sC01YWnPQw+a57L0G5hUxBaPXC2zIoOsk9etLd3erHr+WfFnj2bM9LpT\nTh5kttCBz85o/3n/ZNTZmBiUc4xGMLosM7VBuE/ux1UzCCv6LPPWB3kztFm/dm3mWYnpz8hWOd8X\nfCiHvYbicn5yUM6TkYaZhs2gT1D61WTS8l7GtEVcbXFx2BQb7FYwFM9nZ3Tg3PD3ef4NSk2MEexi\nqQvlvR99yOnG10uAJU15kOXuSYfIudniGd+zDh852pbWWFrSwVde6ToPi8eYDqJXs4sNrdm7v+vj\n5l9tf/+mGeV8sqXK+fpTKx2zZu/+MS7n4Z//Ydf15rbsUlpRXObLrpdfjR/DMjsTfBDm9NVpHuMe\nlPPWh3JjeSnoLJ/OkAezzLs89YDgxWQ6iFZ9SevqwetcCYOnCgKcVCVl8H8vv89d0vyhI9nTtl+o\n42QR3WoG48H7HNZwdRHG0ScLAACgBuZjEHGfuXbOP/G2k0adDQAYuY//ztbCbV/433cNMSc9ck+N\n8PJU00vOcPjSSojgXEGfGPd0evpkap0wnI8hNXt7Xk1EzjEWpGfynZwqPH9lUzQCsez9K2z2VbOa\nqOIF4oNS73lxDWbJRatcsPCU4eoN9+x/yN23dDrFWDQXLqxZp7Pe9r5RZwMARu6uXcWz9J/1tn8x\nxJz0KPV4mfymwmEabCf0nDkkegqyVqb0LSuNwFJ7Dl/Q4czz3nOX7vlqpTPRXAgAAFCDsWguvOii\ni3zHjh2jzgYAoF+pjs91fb5kmopymw4xXrLTceRtl+osNYNiZtqwYcPkNBcCAKZJ54/J0saiYBRX\nfkVAJm1CgqtRN4J1a5A9x7KhU7aH1CBDq6JG6rIyV+m84ZxdFdFcCAAAUANqsgAAA5N6rp97+lFy\nOZM8Zqsd0hN4prvO51ZFtM3LlM8qHN9cTU2HlHP+tnMVVZcUZGgixxZWHcxXsE/bxKyZMlDLPcmW\npwJtZSyZM621Mf1Q8+pVWT3XZJnZZjO738weM7NHzezjcfrJZnavmf04/r6+12sAACZMTkueJ59W\n8Vf2M8rbj5Nau+Z+xef1vMPNWl/xNd1Mrpwva25rnaOVZnJT8pXNbvIV7JPOR/A1DcLXU/TGtB0Q\nS+2TUwYKL9Quee+bZSv48qK8lL0HYTnzzOHN86ZexHAmI21I+oS7XyDprZKuN7MLJN0oaYe7ny9p\nR7wOAACwovTcXOjueyXtjZdfMbPHJZ0haaukS+Pdtkv6jqRP9pVLAMBECJ9yY+GKl7TdlFQMFD4y\nLuxF3rYctvcU7Fd2vjo60k9LbVZTpdeTuY9WkJ49xIL3L6e92JoTmebkoTRbZeWscENQMPKauDsY\nSJ8sMztL0lskPSjptDgAk6TnJZ3W6fhjhw/pZw//n0FkBQAwlorab/I+LfM6b9Wj17O3z/2OcmXv\nc1XZDm+lkXbH3ITlLBV/p540bm1X7kbfowvNbI2kr0u6wd1fDre5J62jecddZ2Y7zWzn/oMv5+0C\nAAAwsfqqyTKzeUUB1q3u/o04+QUz2+jue81so6R9ece6+zZJ26To2YX3ffa3+skKAGBceOHKRKsy\nuG7lqfLMwCr7Zffv5prdHeM5S1a4vdlEWeESOfoZXWiSbpb0uLv/cbDpbknXxMvXSJqAJ5oCAAbG\nPOlX0xpQlhnBF24bA93kpUpYsXK+Ov3L36/4jFXuafZcVe593rXD4zps9/Rrqaqfmqy3S/qQpIfN\nbFec9vuSPiPpDjO7VtIzkj7QxzUAABPHkm+tObOUVAdkP6JS/cz77bbTpyqXS+1DlVZPBjm2wKu8\nB2VdtQq2WclaVf2MLvy/JVe9rNfzAgAATANmfAcADFBmpJe3UruvJpoAk5bfaVT1PSjcr2Ry1GQ6\nid6mpSfIAgAMkCWfV9nHz3R6QPKkPUAZLaNoNe2uu3vZ8Z5/srKZRSriAdEAAAA1oCYLADBQrYm9\nW5OJtv3xH4yJr/BMZYy5cX3fisZUtKUHo2GTJm5vPrVAPVfVEWQBAAbI84d7pWbPLh6ub+P6aY3p\nlTxIPGzszutHGK/QXAgAADBa1GQBAAbG3VLP1M1sjfdppYx4WixAHhTIoFIr3qZgW/dDMwiyAAAD\nEz5+xM1zW1baJnnMGYEI1CpoCywMmSrNclqO5kIAAIAaUJMFABicaEiWpOYjdcqemdM8pvZcASnh\nmNf2hz0Nrj6VIAsAMDhm+QOygkm1aRLEqLVPKVLPdQiyAAAD1aHfe7WDgUEbQY0pfbIAAABqQJAF\nAABWjiHWltJcCAAYjqpPiAbqMILyRU0WAABADQiyAAAAakCQBQAAUAOCLAAAgBoQZAEAANSAIAsA\nAKAGfQdZZjZrZj8ws2/F62eb2YNmttvMvmZmC/1nEwAAYLIMoibr45IeD9Y/K+lz7n6epP2Srh3A\nNQAAACZKX0GWmW2S9D5Jfx6vm6R3Sroz3mW7pCv7uQYAAMAk6rcm6/OSfk/Scrx+iqQD7t6I1/dI\nOiPvQDO7zsx2mtnOQ4sjeGojAABAjXoOsszs1yXtc/eHejne3be5+xZ337JmgWcpAACA6dLPswvf\nLun9ZnaFpFWSTpL0BUnrzGwurs3aJOm5/rMJAAAwWXquyXL3T7n7Jnc/S9LVku5z938l6X5JV8W7\nXSPprr5zCQAAMGHqmCfrk5J+18x2K+qjdXMN1wAAABhr/TQXJtz9O5K+Ey8/JemSQZwXAABgUjHj\nOwAAQA0IsgAAAGpAkAUAAFADgiwAAIAaEGQBAADUgCALAACgBgRZAAAANSDIAgAAqAFBFgAAQA0I\nsgAAAGpAkAUAAFADgiwAAIAaEGQBAADUgCALAACgBgRZAAAANSDIAgAAqAFBFgAAQA0IsgAAAGpA\nkAUAAFADgiwAAIAa9BVkmdk6M7vTzJ4ws8fN7FfN7GQzu9fMfhx/Xz+ozAIAAEyKfmuyviDpHnd/\no6R/JulxSTdK2uHu50vaEa8DAACsKD0HWWa2VtI7JN0sSe6+6O4HJG2VtD3ebbukK/vNJAAAwKTp\npybrbEkvSvqSmf3AzP7czE6QdJq77433eV7SaXkHm9l1ZrbTzHYeWvQ+sgEAADB++gmy5iRdLOmL\n7v4WSa8q0zTo7i4pN4Jy923uvsXdt6xZsD6yAQAAMH76CbL2SNrj7g/G63cqCrpeMLONkhR/39df\nFgEAACZPz0GWuz8v6Vkz+6dx0mWSHpN0t6Rr4rRrJN3VVw4BAAAm0Fyfx39M0q1mtiDpKUkfURS4\n3WFm10p6RtIH+rwGAADAxOkryHL3XZK25Gy6rJ/zAgAATDpmfAcAAKgBQRYAAEANCLIAAABqQJAF\nAABQA4IsAACAGhBkAQAA1IAgCwAAoAYEWQAAADUgyAIAAKgBQRYAAEANCLIAAABqQJAFAABQA4Is\nAACAGhBkAQAA1IAgCwAAoAYEWQAAADUgyAIAAKgBQRYAAEANCLIAAABqQJAFAABQg76CLDP792b2\nqJk9Yma3mdkqMzvbzB40s91m9jUzWxhUZgEAACZFz0GWmZ0h6d9J2uLuF0qalXS1pM9K+py7nydp\nv6RrB5FRAACASdJvc+GcpOPNbE7Sakl7Jb1T0p3x9u2SruzzGgAAABOn5yDL3Z+T9N8l/aOi4Oqg\npIckHXD3RrzbHkln5B1vZteZ2U4z23lo0XvNBgAAwFjqp7lwvaStks6WdLqkEyRdXvV4d9/m7lvc\nfcuaBes1GwAAAGOpn+bCX5P0tLu/6O7HJH1D0tslrYubDyVpk6Tn+swjAADAxOknyPpHSW81s9Vm\nZpIuk/SYpPslXRXvc42ku/rLIgAAwOTpp0/Wg4o6uP+9pIfjc22T9ElJv2tmuyWdIunmAeQTAABg\nosx13qWYu98k6aZM8lOSLunnvAAAAJOOGd8BAABqQJAFAABQA4IsAACAGhBkAQAA1IAgCwAAoAYE\nWQAAADUgyAIAAKgBQRYAAEANCLIAAABqQJAFAABQA4IsAACAGhBkAQAA1KCvB0QDAACMlBeu5DOr\nKydtqMkCAACoATVZAABg8njbQpfHddas8+ryCgmCLAAAMFlcrQhIPTT/eStwstQ5PLtbXwiyAADA\neMrtb1U9qPL4GHOl+2JZlbOkIrme0CcLAACgBtRkAQCA8dPWVldSq5RTyeWS5BYsZ86Wc7qo7qrf\nRsIWgiwAADA58mKtnDTzVnNhHo8DsD5bBEt1bC40s1vMbJ+ZPRKknWxm95rZj+Pv6+N0M7M/MbPd\nZvZDM7u4vqwDAACMryp9sr4s6fJM2o2Sdrj7+ZJ2xOuS9F5J58df10n64mCyCQCYZJ787yocszW4\nVhpMspIi0vlYT746nyK+kHuyaKlqrf6ruDoGWe7+XUm/yCRvlbQ9Xt4u6cog/SseeUDSOjPb2Hcu\nAQATzUrWOiVjBSmLjEwlZSSJkpIvM8nMZNFCsmxmbaeJd0lfaADlsdfRhae5+954+XlJp8XLZ0h6\nNthvT5zWxsyuM7OdZrbz0CJ/vgAAgOnS9xQO7t5TxZ67b3P3Le6+Zc0Cf74AwMrBH9YoYJmvJD2v\nzITNz9kDLbXdgmWXR4MO865Tkp1e9BpkvdBsBoy/74vTn5O0OdhvU5wGAFjxcpph+v0Uw/RKxVWW\n2eC521r1PpmJRINk8/S6PL2aSjQvCPCq6TXIulvSNfHyNZLuCtI/HI8yfKukg0GzIgAAwIrRcZ4s\nM7tN0qWSNpjZHkk3SfqMpDvM7FpJz0j6QLz7tyVdIWm3pNckfaSGPAMAJplTbYWKUkWloPZKikYI\nNtfjxagjezwZabP1sO08aqsk8/yVnnQMstz9gwWbLsvZ1yVd31+WAADTqPtWly6fHdf/o+YwCuH7\nVlhG2h/c3Kk8pYpDEISZpVbjQYmtKeMzm+LU3vDsQgAAgBrwWB0AwEAVjQNrVgdUrxUoe1Zdqk2o\nuxMzuHG85b6P6TY9S002Go6cCFNdHqdbWE7cFbYehs2KmVN0k8FcBFkAgJqkP61yP5o80yRT4fPL\nFX5o5vfTKUWT4vgqfW88Z6n9sOY2d5d8Odip+bBoC1fTT49OBe3ZdsXuEWQBknRc/KPwrl+SFoIf\ni797LPr+0qHh5wkYtCGX81RtQWqLpzq/W2HVQXAGT6e2VipGTH1+WKJuRTWSee+bZdJbva9c0vLc\nrCTp+UvO0/J8VM7NpFN/8JQk6biDhwuOVrJve1566/BHnywAAIAaUJMFSFqeif7e2Lf5FC0fv5Ck\nb3hgXpK0kHsUMFmGX85NZtk6gyi91YUmp4qqYBLK3MYi96TpsNnqk1u5ZdZTyyKGJKiwtML3qfhN\n86BsLc9HNVkvXvR6LS1EZVuzM1r742jazuMOHg4v2Xbawsv3UBtKkAVIarbaPy+pEaSfFH8nyMI0\nGHY596CFpa25sKkoIFIm1ir5fHMPgjcpt09826loPRxbue975aA4HYo3W6U9W+Yyk8Z7UFjaHx8d\nH5RqOqyG5kIAAIAaUJMFABigSjNLFshUMVQ5PDM0P0yrWhOGcZY3tYJntyQjBtsPb6ZbUKyyHedb\nq4W1n6kRiJ3y3EKQBQArxCe++UK08M07crd/4fL1fV/D3bKffvn7FaRa8PlX5bPMw6Au+wlZFljR\nJ2t8VJmPoW2Hkjc3r4nal1ujWHP6VlnHwL63KJ3mQgAAgBpQkwVMkBs++qPS7Z//X28YUk4waT5+\nz/5K+/Rbm2WZmqVmp/Sq9QDu+d2O053lg07K3mpeTM8AXnleS4ybsLN74UTsZU8DaH7zYDo2S8pN\nMulteJ2i5WSxQvVsDoIsYEJ0CrCa+xBooW777/lF4bb1YZBWqU+VUhOThoe196zKm84h58O2Wjsj\nRqXvptpmUF1+IvNmEF7S6pgsB2UrDK6Sfn6tUaxWdQJc0VwIAABQC2qyAACVlNVg5YkeCt2cqKhg\nisdMK2BqDqvwqTrBZKL5jwMOLxzWdFBlNXZK35KiWqLizurtYw3jIzJNf2393fPm4HJPF668p/10\nMSnp1ARZa1avliRdcO45SVpjaUk/fPJHyTIwDPOrTpAkrd10vg69uEeSdOTgz0eZJWCI8j8kLfWE\nwuLeNeamE1YfLyn6fd5smgl/nx9rLBUOKGw7o+fnh9BrvHSe9KM8YLa2hWil+fZbtqk5DO7DaeYz\nxWV+1RpJ0okbz9Ghnz8nSVp8pfofGzQXAgAA1GBqarLmZqNnFa098cQk7Vij0VUHNWBe6b88eik9\n6858oyTpPf/xNn1v+x9Jkn70t7f2nTdg0qQeF5iqiCgaqWVyuebD3+fNmqxGo3WcKZh4NH3NVBOh\nKzNZac8vBTXr+NZkp1/zggESQc2lLS+3dvLlTHVX+ooFxSn5ff6uP7hVD27/T5Kkn9z3F51ym5ia\nIAvox2z8/bxM+nw/JyXAxxj5wuXrO07jMIjJSKX0xOuth/1m+2S1f6x6/LDn5Dly4Yivtgkry2YC\nDz52CaymQ+Z9LG5+ds3Eq6sOHtbc0aV4f2mmES/HcziED5XO7WdlJrf4z+7ZGc3ED1jvNKoxdQrv\n4anSg/amN57vd/zZ5/s6R/MvnxPXnJCkubsOvnJIkrQ8Bq8TY2wm/qHZuL61LEn7Dkbfjzbajymw\nsDqqTT35rDfp5b1PS5Je2//CQLJ54cu/Urr9kZMeHMh1MKX6LOd/+Y73dbzEVX/3rUz/mrD2ScGW\nVnoyzD7eZ34u+n1+0gkntCoi3HXw0KvJcjKMv2Mnan73T4ZOvbKsNYZCUvj08fApATJJs1EwdPSk\n47XcnBvLTPOvHJYkzSw2MiUjuKZb62fDl7VwfPT49PVnXaCXfxb9Pj+8/3m9+dL3P+TuWzq9qo59\nsszsFjPbZ2aPBGn/zcyeMLMfmtk3zWxdsO1TZrbbzJ40s/d0Oj8AAMA06liTZWbvkHRI0lfc/cI4\n7d2S7nP3hpl9VpLc/ZNmdoGk2yRdIul0SX8r6Q3uXjq076KLLvIdO3b0/WIAAPX7nxtOKdz2sRdf\nCse6p7pHVZLprtXqRpOplerQ0oNplK3taq/9sjA5tWtOYs4UDh7048s2CoZPL3jdqadWqsnq2CfL\n3b9rZmdl0v4mWH1A0lXx8lZJt7v7UUlPm9luRQHX/+t0HQDAZPjYz18q3hh0hKoc/ASfldkGvvTn\nYM6cWwRYK0jRRFfplNQjcHLnvLLUNw/mxrLgUVDhyA03JcHXTBdR/SCmcPgtSX8VL58h6dlg2544\nrY2ZXWdmO81s50svlfzAAgAATKC+giwz+wNJDUldj093923uvsXdt5xySnHVMwBgcrQ6u7c6JZu1\nUqIvS83anprB3ZvPnMtu89ZXctL0uH7G864MSbkJikGzoKTLSnhQUH7yqj+TTdEI12T6p+C85h4N\n0uhi5HjPUziY2b+W9OuSLvNWx67nJG0OdtsUpwEAVphwmH32cykZBp/3tB1vhWq5fWky50nvEk7l\nzRxZky9oukunpKS25o5iLQ6M2kYn5ubCe5qVp6eaLDO7XNLvSXq/u78WbLpb0tVmdpyZnS3pfEnf\n6+UaAAAAk6xjTZaZ3SbpUkkbzGyPpJskfUrScZLujavUHnD333b3R83sDkmPKWpGvL7TyEIAwPTI\nPDou9VDdVq1WdkhgZjRXYZVBq/bLg/qMwkoqhh1Oj7gZL1psjfKT0uUp7N8eLXeoompuK6zBai3k\nPrCggyqjCz+Yk3xzyf6flvTp6lkAAEyP1sdcNIN7zh5twU/4QdhdYBQ29RBSTbmioNk99d7njk7N\nBvnpE5SsNQ9qbbAuShoPiAYAAKgBzy4EAAxOtoN7hT/608+CS58gt2Um07rjqRXqs6ZV0TvrFWpA\nXUqaodsbmjPVoQVP+GlVgA1hdCEAAO2C/lZBX5n2fliBbHKnz7DMeZi6AYWloLRw5AX36edqKkjt\nBc2FAAAANaAmCwAwONlnv1nYCT54LE5qDqtMzVc4uVa4TWpfTj0uBStG0fsejDw1WbpFMNknp/wF\nM5x6WPuad3gX2ez4gOhhMLMXJb0q6eejzsuY2yDuURnuT2fco864R+W4P51xjzqb9Hv0enc/tdNO\nYxFkSZKZ7azyROuVjHtUjvvTGfeoM+5ROe5PZ9yjzlbKPaJPFgAAQA0IsgAAAGowTkHWtlFnYAJw\nj8pxfzrjHnXGPSrH/emMe9TZirhHY9MnCwAAYJqMU00WAADA1CDIAgAAqMHIgywzu9zMnjSz3WZ2\n46jzMy7M7Kdm9rCZ7TKznXG8F+odAAAEuUlEQVTayWZ2r5n9OP6+ftT5HCYzu8XM9pnZI0Fa7j2x\nyJ/E5eqHZnbx6HI+PAX36A/N7Lm4LO0ysyuCbZ+K79GTZvae0eR6eMxss5ndb2aPmdmjZvbxOJ1y\npNL7QxmKmdkqM/uemf1DfI/+KE4/28wejO/F18xsIU4/Ll7fHW8/a5T5H4aSe/RlM3s6KEcXxenT\n+3Pm7iP7kjQr6SeSzpG0IOkfJF0wyjyNy5ekn0rakEn7r5JujJdvlPTZUedzyPfkHZIulvRIp3si\n6QpJf6Voot63Snpw1Pkf4T36Q0n/IWffC+KfueMknR3/LM6O+jXUfH82Sro4Xj5R0o/i+0A5Kr8/\nlKHWazZJa+LleUkPxmXjDklXx+l/Kunfxsu/I+lP4+WrJX1t1K9hhPfoy5Kuytl/an/ORl2TdYmk\n3e7+lLsvSrpd0tYR52mcbZW0PV7eLunKEeZl6Nz9u5J+kUkuuidbJX3FIw9IWmdmG4eT09EpuEdF\ntkq63d2PuvvTknYr+pmcWu6+193/Pl5+RdLjks4Q5UhS6f0pshLLkLv7oXh1Pv5ySe+UdGecni1D\nzbJ1p6TLLHxeyxQquUdFpvbnbNRB1hmSng3W96j8B3olcUl/Y2YPmdl1cdpp7r43Xn5e0mmjydpY\nKbonlK20j8bV8LcEzcwr+h7FzTZvUfRXNuUoI3N/JMpQwsxmzWyXpH2S7lVUg3fA3RvxLuF9SO5R\nvP2gpFOGm+Phy94jd2+Wo0/H5ehzZnZcnDa15WjUQRaK/XN3v1jSeyVdb2bvCDd6VMfK/BsB7kmh\nL0o6V9JFkvZK+h+jzc7omdkaSV+XdIO7vxxuoxzl3h/KUMDdl9z9IkmbFNXcvXHEWRo72XtkZhdK\n+pSie/XLkk6W9MkRZnEoRh1kPSdpc7C+KU5b8dz9ufj7PknfVPSD/EKzCjX+vm90ORwbRfeEshVz\n9xfiX3jLkv5MreacFXmPzGxeUQBxq7t/I06mHMXy7g9lKJ+7H5B0v6RfVdTENRdvCu9Dco/i7Wsl\nvTTkrI5McI8uj5uj3d2PSvqSVkA5GnWQ9X1J58ejMhYUdQq8e8R5GjkzO8HMTmwuS3q3pEcU3Ztr\n4t2ukXTXaHI4Voruyd2SPhyPWnmrpINBc9CKkunb8BuKypIU3aOr49FPZ0s6X9L3hp2/YYr7wtws\n6XF3/+NgE+VIxfeHMtRiZqea2bp4+XhJ71LUd+1+SVfFu2XLULNsXSXpvri2dGoV3KMngj9kTFGf\ntbAcTeXP2VznXerj7g0z+6ikv1Y00vAWd390lHkaE6dJ+mbcN3JO0l+4+z1m9n1Jd5jZtZKekfSB\nEeZx6MzsNkmXStpgZnsk3STpM8q/J99WNGJlt6TXJH1k6BkegYJ7dGk8VNoVjVr9N5Lk7o+a2R2S\nHpPUkHS9uy+NIt9D9HZJH5L0cNxfRJJ+X5SjpqL780HKUGKjpO1mNquoouIOd/+WmT0m6XYz+y+S\nfqAoWFX8/atmtlvRoJSrR5HpISu6R/eZ2amKRhHukvTb8f5T+3PGY3UAAABqMOrmQgAAgKlEkAUA\nAFADgiwAAIAaEGQBAADUgCALAACgBgRZAAAANSDIAgAAqMH/B5yyMnuKi/egAAAAAElFTkSuQmCC\n","text/plain":["
"]},"metadata":{"tags":[]}}]},{"cell_type":"code","metadata":{"id":"eSYdU54DOiCy","colab_type":"code","cellView":"form","outputId":"ee26eefb-09f0-4265-bee2-936426208975","executionInfo":{"status":"ok","timestamp":1571160788156,"user_tz":-60,"elapsed":221641,"user":{"displayName":"Sebastian Borgeaud","photoUrl":"","userId":"12373152407193217991"}},"colab":{"base_uri":"https://localhost:8080/","height":821}},"source":["#@title Display the video (ground truth, keypoints, reconstruction)\n","mp4 = open('collage.mp4','rb').read()\n","data_url = \"data:video/mp4;base64,\" + b64encode(mp4).decode()\n","\n","HTML(\"\"\"\n","\n","\"\"\" % data_url)"],"execution_count":9,"outputs":[{"output_type":"execute_result","data":{"text/html":["\n","\n"],"text/plain":[""]},"metadata":{"tags":[]},"execution_count":9}]}]}