<?xml version="1.0" encoding="UTF-8"?><rss version="2.0" xmlns:content="http://purl.org/rss/1.0/modules/content/"><channel><title>Satyajit Sarangi</title><description>Systems engineer writing about AI inference infrastructure: inference engines, tool-call parsing, LoRA, constrained decoding, and compilers.</description><link>https://ssarangi.github.io/</link><item><title>Udacity&apos;s Self Driving Car - Vehicle Detection</title><link>https://ssarangi.github.io/posts/udacityselfdrivingcar-vehicle-detection/</link><guid isPermaLink="true">https://ssarangi.github.io/posts/udacityselfdrivingcar-vehicle-detection/</guid><description>This post talks about vehicle detection from video stream</description><pubDate>Mon, 22 May 2017 00:00:00 GMT</pubDate><content:encoded>&lt;h2 id=&quot;vehicle-detection-project&quot;&gt;Vehicle Detection Project&lt;/h2&gt;
&lt;p&gt;Vehicle detection is a quite highly researched area with open datasets like KITTI and others from Udacity all over the web. Realtime models like Yolo to better accuracy models like R-CNN to more complicated models have made this topic more and more accessible with pre-trained models. This project was the 5th and last project from the Term 1 of Udacity’s Self Driving Cars.&lt;/p&gt;
&lt;p&gt;This project was focused towards using Support Vector Machines, Decision Trees along with a host of Computer Vision Principles. However, after trying out and experimenting with all those techniques I decided to use a Deep Learning approach to this problem because of quality reasons.&lt;/p&gt;
&lt;p&gt;I could not get a model reliable enough along with the heatmap approach to filter out false positives. Thus the deviation to using a Convolutional Neural Network for this project.&lt;/p&gt;
&lt;p&gt;The goal of this project is to recognize vehicles from a live video capture.&lt;/p&gt;
&lt;h2 id=&quot;high-level-approach&quot;&gt;High Level Approach&lt;/h2&gt;
&lt;p&gt;The high level algorithm of this project was as follows.&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Explore the dataset provided by Udacity.&lt;/li&gt;
&lt;li&gt;Train a CNN on this dataset.&lt;/li&gt;
&lt;li&gt;Process Each frame of the video clip&lt;/li&gt;
&lt;li&gt;Use a sliding window approach to search for vehicles in the video frame&lt;/li&gt;
&lt;li&gt;Predict the presence of a vehicle in each sliding window&lt;/li&gt;
&lt;li&gt;Filter out false positives with a Heatmap approach.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id=&quot;training-dataset&quot;&gt;Training dataset&lt;/h2&gt;
&lt;p&gt;The training dataset by Udacity was a balanced dataset with most of it coming from the GTI / KITTI Dataset. The images we classified into either vehicles and non-vehicles. One important feature of this dataset was for for vehicles it provided examples of different cars, different orientations (left, right, back) etc.&lt;/p&gt;
&lt;p&gt;[[ Images for the car from different orientations ]]&lt;/p&gt;
&lt;p&gt;For the non-vehicles also different categories of images were present. This included trees, roads, sky, edge of roads etc.&lt;/p&gt;
&lt;p&gt;All in all, however it was a balanced dataset.&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Vehicles: 8792&lt;/li&gt;
&lt;li&gt;Non Vehicles: 8968&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id=&quot;abandoning-the-hog--svm-approach&quot;&gt;Abandoning the HOG / SVM approach&lt;/h2&gt;
&lt;p&gt;I tried the HOG + color histogram + spatial features approach with both SVM &amp;#x26; Decision Tree. The SVM &amp;#x26; Decision Tree, when trained on the training data was giving ac accuracy of about 97 to 98% however, running on the actual video clip was giving a lot more false positives. Neither playing with different HOG parameters nor the heatmap approach helped with the false positive elimination and thus I abandoned that approach.&lt;/p&gt;
&lt;h2 id=&quot;convolutional-neural-network&quot;&gt;Convolutional Neural Network&lt;/h2&gt;
&lt;p&gt;Since we have dealt with CNN’s in our past project, this was yet another good opportunity to try out deep learning for this project. I used Keras and wrote a simple model, very similar to Lenet but having 2 more extra convolutional layers.&lt;/p&gt;
&lt;p&gt;The network was mostly inspired by this blog post on the &lt;a href=&quot;https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html&quot;&gt;Keras Blog&lt;/a&gt;&lt;/p&gt;
&lt;h3 id=&quot;architecture&quot;&gt;Architecture&lt;/h3&gt;
&lt;p&gt;&lt;img src=&quot;https://ssarangi.github.io/img/blog/vehicle_detection/model.png&quot; alt=&quot;alt text&quot; title=&quot;Custom Model&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; width=&quot;588&quot; height=&quot;2176&quot;&gt;&lt;/p&gt;
&lt;h3 id=&quot;model-description&quot;&gt;Model Description&lt;/h3&gt;
&lt;p&gt;The model is a simple 4 layer ConvNet with dropouts added in between since dropout also helps reduce overfitting, by preventing a layer from seeing twice the exact same pattern, thus acting in a way analoguous to data augmentation &lt;a href=&quot;https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html&quot;&gt;Keras Blog&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;The first 4 conv layers increase in the number of inputs they take to slowly learn the parameters. The use of 4 layers was to make sure that the network learns enough characteristics to distinguish vehicle and non-vehicle images with minimal false positives.&lt;/p&gt;
&lt;h3 id=&quot;data-augmentation&quot;&gt;Data Augmentation&lt;/h3&gt;
&lt;p&gt;Since the dataset isn’t very big, overfitting is a concern and hence I decided to perform training data augmentation using the Keras generator. Keras generator lets us augment the training data at runtime with a very simple to use api. The following lines of code show how the Image Generator is used to augment the training data.&lt;/p&gt;
&lt;pre class=&quot;astro-code astro-code-themes github-light github-dark-dimmed&quot; style=&quot;--shiki-light:#24292e;--shiki-dark:#adbac7;--shiki-light-bg:#fff;--shiki-dark-bg:#22272e; overflow-x: auto;--file-name-offset: -0.75rem;&quot; tabindex=&quot;0&quot; data-language=&quot;python&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;generator &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; ImageDataGenerator(&lt;/span&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;featurewise_center&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;True&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;                               samplewise_center&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;False&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;                               featurewise_std_normalization&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;False&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;                               samplewise_std_normalization&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;False&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;                               zca_whitening&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;False&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;                               rotation_range&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;20&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;.,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;                               width_shift_range&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;0.4&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;                               height_shift_range&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;0.4&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;                               shear_range&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;0.2&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;                               zoom_range&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;0.2&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;                               channel_shift_range&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;0.1&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;                               fill_mode&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&apos;nearest&apos;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;                               horizontal_flip&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;True&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;                               vertical_flip&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;False&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;                               rescale&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;1.2&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;                               preprocessing_function&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;None&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;)&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;&lt;img src=&quot;https://ssarangi.github.io/img/blog/vehicle_detection/keras_augmented.png&quot; alt=&quot;alt text&quot; title=&quot;Keras Augmented Training Dataset&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; width=&quot;500&quot; height=&quot;500&quot;&gt;&lt;/p&gt;
&lt;h3 id=&quot;network-performance&quot;&gt;Network Performance&lt;/h3&gt;
&lt;p&gt;&lt;img src=&quot;https://ssarangi.github.io/img/blog/vehicle_detection/validation_acc.png&quot; alt=&quot;alt text&quot; title=&quot;Validation Accuracy vs Accuracy over epochs&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; width=&quot;540&quot; height=&quot;377&quot;&gt;
&lt;img src=&quot;https://ssarangi.github.io/img/blog/vehicle_detection/validation_loss.png&quot; alt=&quot;alt text&quot; title=&quot;Validation Loss vs Loss over epochs&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; width=&quot;540&quot; height=&quot;377&quot;&gt;
As can be seen over 100 epochs, the network does overfit quite a bit. This would need more tuning and possible addition of L2 normalization factors&lt;/p&gt;
&lt;h2 id=&quot;inception-model&quot;&gt;Inception Model&lt;/h2&gt;
&lt;p&gt;I was curious to try out a smaller inception model since inception models let you try different convolutional layers without making them very deep. The
advantage is that the layers don’t have to be as deep and yet learn various features from the images due to the use of different Convolution sizes.&lt;/p&gt;
&lt;h3 id=&quot;architecture-1&quot;&gt;Architecture&lt;/h3&gt;
&lt;p&gt;The chosen architecture was very simple. It consists of 3 parallel layers, 2 Conv and 1 Maxpool layers.
1st parallel layer consists of 2 Conv Layers of (1, 1) &amp;#x26; (3, 3)
2nd parallel layer consists of 2 Conv Layers of (1, 1) &amp;#x26; (5, 5)
3rd parallel layer is the maxpool layer.
&lt;img src=&quot;https://ssarangi.github.io/img/blog/vehicle_detection/inception_model.png&quot; alt=&quot;alt text&quot; title=&quot;Inception Model&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; width=&quot;1346&quot; height=&quot;1069&quot;&gt;&lt;/p&gt;
&lt;h3 id=&quot;network-performance-1&quot;&gt;Network Performance&lt;/h3&gt;
&lt;p&gt;This network was comparatively slower than the Custom network and hence I ran it for 30 epochs. Performance wise it reached about 93% accuracy but started overfitting after about 85%.
I would have to tune the network more for it to work better with vehicle detection. This network generated more false positives as compared to the custom network.
&lt;img src=&quot;https://ssarangi.github.io/img/blog/vehicle_detection/inception_validation_acc.png&quot; alt=&quot;alt text&quot; title=&quot;Validation Accuracy vs Accuracy over epochs&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; width=&quot;535&quot; height=&quot;377&quot;&gt;
&lt;img src=&quot;https://ssarangi.github.io/img/blog/vehicle_detection/inception_validation_loss.png&quot; alt=&quot;alt text&quot; title=&quot;Validation Loss vs Loss over epochs&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; width=&quot;527&quot; height=&quot;377&quot;&gt;&lt;/p&gt;
&lt;h2 id=&quot;prediction&quot;&gt;Prediction&lt;/h2&gt;
&lt;p&gt;Once the network is saved and the model is saved the prediction pipeline begins. The prediction pipeline has 2 important steps.&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Sliding Windows to identify regions of images&lt;/li&gt;
&lt;li&gt;Heatmap to filter out false positives.&lt;/li&gt;
&lt;/ul&gt;
&lt;h3 id=&quot;sliding-windows&quot;&gt;Sliding Windows&lt;/h3&gt;
&lt;p&gt;The sliding window approach basically is an algorithm where we decide how to take regions from the image and then check whether they have a vehicle in them or not.&lt;/p&gt;
&lt;h4 id=&quot;choosing-different-window-sizes&quot;&gt;Choosing different Window Sizes&lt;/h4&gt;
&lt;p&gt;I chose 3 different window sizes to cover the entire region of interest. The region of interest was the area of the image where the cars would appear if it was going on a flat plane. This is an obvious assumption and wouldn’t work in places like San Francisco where hilly roads are present and hence there would be vehicles outside the region of interest.
&lt;img src=&quot;https://ssarangi.github.io/img/blog/vehicle_detection/roi.png&quot; alt=&quot;alt text&quot; title=&quot;Region of Interest&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; width=&quot;672&quot; height=&quot;395&quot;&gt;&lt;/p&gt;
&lt;p&gt;The 3 different window sizes used were&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;(30, 30)&lt;/li&gt;
&lt;li&gt;(60, 60)&lt;/li&gt;
&lt;li&gt;(90, 90)&lt;/li&gt;
&lt;li&gt;(120, 120)&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;These window sizes were chosen based on eyeballing different car sizes across the image.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://ssarangi.github.io/img/blog/vehicle_detection/sliding_window1.png&quot; alt=&quot;alt text&quot; title=&quot;Sliding Window (30x30)&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; width=&quot;878&quot; height=&quot;511&quot;&gt;
&lt;img src=&quot;https://ssarangi.github.io/img/blog/vehicle_detection/sliding_window2.png&quot; alt=&quot;alt text&quot; title=&quot;Sliding Window (60x60)&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; width=&quot;878&quot; height=&quot;511&quot;&gt;
&lt;img src=&quot;https://ssarangi.github.io/img/blog/vehicle_detection/sliding_window3.png&quot; alt=&quot;alt text&quot; title=&quot;Sliding Window (90x90)&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; width=&quot;878&quot; height=&quot;511&quot;&gt;
&lt;img src=&quot;https://ssarangi.github.io/img/blog/vehicle_detection/sliding_window4.png&quot; alt=&quot;alt text&quot; title=&quot;Sliding Window (120x120)&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; width=&quot;878&quot; height=&quot;511&quot;&gt;&lt;/p&gt;
&lt;h4 id=&quot;heatmap-for-filtering-false-positives&quot;&gt;HeatMap for filtering False positives&lt;/h4&gt;
&lt;p&gt;A heatmap approach is used to filter out the false positives. The heatmap is a 2D array with a single channel which helps in eliminating false positives from the frame. The technique is rather intuitive. The 2D array is the same size as the image. Once the prediction from the sliding windows is obtained all the windows which are identified as having a vehicle are incremented by 1. The “hot windows” are the ones which contain value greater than a certain threshold. Once the threshold is crossed these areas are reliably marked as being areas with vehicles in them. There is also a frame averaging done to keep the history of a certain number of previous frame’s heatmaps and these are then accumulated with the current heatmap.
For this project I average the last 3 frames and use a threshold of 1. The output video images show the heatmap.&lt;/p&gt;
&lt;h4 id=&quot;multithreading-didnt-work-&quot;&gt;Multithreading (didn’t work !!!)&lt;/h4&gt;
&lt;p&gt;Using multithreading for prediction across windows proved to be tricky. This was because&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Prediction in a thread meant that the GPU had to make a copy of the model everytime which kept on failing.&lt;/li&gt;
&lt;li&gt;Just performing Cropping on the threads while performing the prediction on the main thread proved to be slow since more time was spent on setting up and tearing down threads than the cropping process itself.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id=&quot;output-prediction-on-test-images-custom-network&quot;&gt;Output Prediction on Test Images (Custom Network)&lt;/h2&gt;
&lt;p&gt;&lt;img src=&quot;https://ssarangi.github.io/img/blog/vehicle_detection/test1_custom.png&quot; alt=&quot;alt text&quot; title=&quot;Custom Network Prediction&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; width=&quot;2000&quot; height=&quot;1218&quot;&gt;
&lt;img src=&quot;https://ssarangi.github.io/img/blog/vehicle_detection/test2_custom.png&quot; alt=&quot;alt text&quot; title=&quot;Custom Network Prediction&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; width=&quot;2000&quot; height=&quot;1218&quot;&gt;
&lt;img src=&quot;https://ssarangi.github.io/img/blog/vehicle_detection/test3_custom.png&quot; alt=&quot;alt text&quot; title=&quot;Custom Network Prediction&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; width=&quot;2000&quot; height=&quot;1218&quot;&gt;
&lt;img src=&quot;https://ssarangi.github.io/img/blog/vehicle_detection/test4_custom.png&quot; alt=&quot;alt text&quot; title=&quot;Custom Network Prediction&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; width=&quot;2000&quot; height=&quot;1218&quot;&gt;
&lt;img src=&quot;https://ssarangi.github.io/img/blog/vehicle_detection/test5_custom.png&quot; alt=&quot;alt text&quot; title=&quot;Custom Network Prediction&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; width=&quot;2000&quot; height=&quot;1218&quot;&gt;
&lt;img src=&quot;https://ssarangi.github.io/img/blog/vehicle_detection/test6_custom.png&quot; alt=&quot;alt text&quot; title=&quot;Custom Network Prediction&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; width=&quot;2000&quot; height=&quot;1218&quot;&gt;&lt;/p&gt;
&lt;h2 id=&quot;output-prediction-on-test-images-inception-network&quot;&gt;Output Prediction on Test Images (Inception Network)&lt;/h2&gt;
&lt;p&gt;&lt;img src=&quot;https://ssarangi.github.io/img/blog/vehicle_detection/test1_inception.png&quot; alt=&quot;alt text&quot; title=&quot;Inception Network Prediction&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; width=&quot;2000&quot; height=&quot;1218&quot;&gt;
&lt;img src=&quot;https://ssarangi.github.io/img/blog/vehicle_detection/test2_inception.png&quot; alt=&quot;alt text&quot; title=&quot;Inception Network Prediction&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; width=&quot;2000&quot; height=&quot;1218&quot;&gt;
&lt;img src=&quot;https://ssarangi.github.io/img/blog/vehicle_detection/test3_inception.png&quot; alt=&quot;alt text&quot; title=&quot;Inception Network Prediction&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; width=&quot;2000&quot; height=&quot;1218&quot;&gt;
&lt;img src=&quot;https://ssarangi.github.io/img/blog/vehicle_detection/test4_inception.png&quot; alt=&quot;alt text&quot; title=&quot;Inception Network Prediction&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; width=&quot;2000&quot; height=&quot;1218&quot;&gt;
&lt;img src=&quot;https://ssarangi.github.io/img/blog/vehicle_detection/test5_inception.png&quot; alt=&quot;alt text&quot; title=&quot;Inception Network Prediction&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; width=&quot;2000&quot; height=&quot;1218&quot;&gt;
&lt;img src=&quot;https://ssarangi.github.io/img/blog/vehicle_detection/test6_inception.png&quot; alt=&quot;alt text&quot; title=&quot;Inception Network Prediction&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; width=&quot;2000&quot; height=&quot;1218&quot;&gt;&lt;/p&gt;
&lt;h2 id=&quot;project-video-custom-network&quot;&gt;Project Video (Custom Network)&lt;/h2&gt;
&lt;p&gt;&lt;video src=&quot;https://ssarangi.github.io/img/blog/vehicle_detection/custom_project_video.mp4&quot; controls preload=&quot;none&quot; playsinline style=&quot;max-width:100%;height:auto&quot; aria-label=&quot;alt text&quot;&gt;&lt;/video&gt;&lt;/p&gt;
&lt;h2 id=&quot;project-video-inception-network&quot;&gt;Project Video (Inception Network)&lt;/h2&gt;
&lt;p&gt;&lt;video src=&quot;https://ssarangi.github.io/img/blog/vehicle_detection/inception_project_video.mp4&quot; controls preload=&quot;none&quot; playsinline style=&quot;max-width:100%;height:auto&quot; aria-label=&quot;alt text&quot;&gt;&lt;/video&gt;&lt;/p&gt;
&lt;h2 id=&quot;future-work&quot;&gt;Future work&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;The models need more tuning for getting good accuracy&lt;/li&gt;
&lt;li&gt;The sliding window technique needs to be more finetuned since right now its hardcoded to a few fixed sizes and hence the detection is highly sensitive to these sizes.&lt;/li&gt;
&lt;li&gt;The prediction has to happen in real time and hence needs to be tunes to predict fast.&lt;/li&gt;
&lt;/ul&gt;</content:encoded><category>python</category><category>keras</category></item><item><title>Udacity&apos;s Self Driving Car - Advanced Lane Finding Algorithm</title><link>https://ssarangi.github.io/posts/udacityselfdrivingcar-advanced-lane-finding/</link><guid isPermaLink="true">https://ssarangi.github.io/posts/udacityselfdrivingcar-advanced-lane-finding/</guid><description>This post talks about the Advanced Lane Finding algorithm for extracting lane lines</description><pubDate>Sun, 07 May 2017 00:00:00 GMT</pubDate><content:encoded>&lt;h2 id=&quot;robust-lane-finding-algorithm&quot;&gt;Robust Lane Finding Algorithm&lt;/h2&gt;
&lt;h2 id=&quot;introduction&quot;&gt;Introduction&lt;/h2&gt;
&lt;p&gt;Lane finding is an important algorithm for autonomous vehicles especially in high traffic areas where the margin of error is very low. Keeping true to the lane is important for the car and this problem has many implications. Udacity’s first project for lane finding from the Self Driving Car Nanodegree was an excellent introduction to this topic but it quickly made me realize how fragile the whole system was and how quickly the algorithm starts to fail. This was the 2nd project which explores the topic further by using some more advanced topics as compared to what was presented in the first project.&lt;/p&gt;
&lt;p&gt;The project revolves around finding lanes by a camera which is mounted at the center of the car. The code is in lane_line_detection.py and the code can be run with the following command line.&lt;/p&gt;
&lt;pre class=&quot;astro-code astro-code-themes github-light github-dark-dimmed&quot; style=&quot;--shiki-light:#24292e;--shiki-dark:#adbac7;--shiki-light-bg:#fff;--shiki-dark-bg:#22272e; overflow-x: auto;--file-name-offset: -0.75rem;&quot; tabindex=&quot;0&quot; data-language=&quot;plaintext&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span&gt;python lane_line_detection.py --pipeline video --file project_video.mp4 --frames 3&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The project also consists of other commands line options which can be used to control the behavior of the project.&lt;/p&gt;
&lt;h2 id=&quot;algorithm-overview&quot;&gt;Algorithm Overview&lt;/h2&gt;
&lt;p&gt;The algorithm starts with first performing offline steps i.e. performing Camera Calibration. Lens typically have distortions and it is necessary to perform distortion correction for the lens before the images from the camera can be further processed.&lt;/p&gt;
&lt;h3 id=&quot;offline-camera-calibration-steps&quot;&gt;Offline Camera Calibration Steps:&lt;/h3&gt;
&lt;p&gt;&lt;img src=&quot;https://ssarangi.github.io/img/blog/advanced_lane_finding/calibration_flow.png&quot; alt=&quot;alt text&quot; title=&quot;Calibration Flow&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; width=&quot;960&quot; height=&quot;540&quot;&gt;&lt;/p&gt;
&lt;p&gt;The diagram above shows a brief overview of how the camera calibration is performed. Multiple images of a known size chess board is first used. Objects points are passed in which get mapped to the images to eventually find the corner points on them. As these are found, a matrix is accumulated which essentially is the camera calibration matrix.&lt;/p&gt;
&lt;p&gt;This matrix is multiplied to every incoming image to undistort it.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://ssarangi.github.io/img/blog/advanced_lane_finding/chess_board_undistorted.png&quot; alt=&quot;alt text&quot; title=&quot;Chess Board Undistorted&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; width=&quot;1000&quot; height=&quot;500&quot;&gt;&lt;/p&gt;
&lt;h3 id=&quot;real-time-processing-steps&quot;&gt;Real time Processing Steps:&lt;/h3&gt;
&lt;p&gt;&lt;img src=&quot;https://ssarangi.github.io/img/blog/advanced_lane_finding/real_time_algo.png&quot; alt=&quot;alt text&quot; title=&quot;Real Time Processing Flow&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; width=&quot;960&quot; height=&quot;540&quot;&gt;&lt;/p&gt;
&lt;p&gt;The steps shown here represent the overall algorithm which is used to find out the lane lines. These have to run in real time so it is very important to keep in mind the processing cost of any associated steps that are there and any step that could be parallelized.&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Initial Input Image provided&lt;/li&gt;
&lt;li&gt;Un-distort based on Camera Calibration&lt;/li&gt;
&lt;li&gt;Perform Saturation Thresholding&lt;/li&gt;
&lt;li&gt;Perform Color thresholding&lt;/li&gt;
&lt;li&gt;Merge the 2 thresholded images to extract the white &amp;#x26; yellow lane markings on the road&lt;/li&gt;
&lt;li&gt;Define Region of Interest&lt;/li&gt;
&lt;li&gt;Perform Perspective Transformation to convert to bird’s eye view.&lt;/li&gt;
&lt;li&gt;Find areas of interest on the images by figuring out pixel intensities and fit a polynomial to it&lt;/li&gt;
&lt;li&gt;Mark the region on birds-eye view and perform inverse transform to project on initial undistorted image&lt;/li&gt;
&lt;/ol&gt;
&lt;h4 id=&quot;undistortion&quot;&gt;Undistortion&lt;/h4&gt;
&lt;p&gt;The undistortion is performed by multiplying the transformation matrix we obtained from the camera calibration. The image below shows the result from the un-distortion from the camera calibration matrix.&lt;/p&gt;
&lt;p&gt;The class CameraCalibration has methods which store the calibration file once its calibrated and also has the methods to load and calibrate the camera.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://ssarangi.github.io/img/blog/advanced_lane_finding/input_image_undistorted.png&quot; alt=&quot;alt text&quot; title=&quot;Undistored Input Image&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; width=&quot;1000&quot; height=&quot;500&quot;&gt;&lt;/p&gt;
&lt;h4 id=&quot;saturation-thresholding&quot;&gt;Saturation Thresholding&lt;/h4&gt;
&lt;p&gt;Yellow color lanes are best extracted by doing thresholding on the saturation images. So once we have the un-distorted images we convert it to HLS image. I use a thresholding values of (90, 255) to extract the yellow lines from the image. The image below shows both the saturation channel and the yellow line extracted.&lt;/p&gt;
&lt;p&gt;The Thresholder class is responsible for having all the methods for thresholding.
Thresholder::simple_threshold is used for doing threshold on the saturation channel.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://ssarangi.github.io/img/blog/advanced_lane_finding/saturation_threshold.png&quot; alt=&quot;alt text&quot; title=&quot;Saturation Thresholding&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; width=&quot;1000&quot; height=&quot;500&quot;&gt;&lt;/p&gt;
&lt;h4 id=&quot;color-thresholding--merging&quot;&gt;Color Thresholding &amp;#x26; Merging&lt;/h4&gt;
&lt;p&gt;I tried doing a combined thresholding to extract the white lane lines but the results weren’t as well. So I do a Color thresholding by first converting the image to HSV. I use different ranges of whites to extract the lanes. The code below shows the thresholds used to compute the color thresholds.&lt;/p&gt;
&lt;pre class=&quot;astro-code astro-code-themes github-light github-dark-dimmed&quot; style=&quot;--shiki-light:#24292e;--shiki-dark:#adbac7;--shiki-light-bg:#fff;--shiki-dark-bg:#22272e; overflow-x: auto;--file-name-offset: -0.75rem;&quot; tabindex=&quot;0&quot; data-language=&quot;python&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;sensitivity_1 &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; 68&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;white &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; cv2.inRange(hsv, (&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;0&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;, &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;0&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;, &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;255&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; sensitivity_1), (&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;255&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;, &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;20&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;, &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;255&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;))&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;sensitivity_2 &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; 60&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;hsl &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; cv2.cvtColor(img, cv2.&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;COLOR_RGB2HLS&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;white_2 &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; cv2.inRange(hsl, (&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;0&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;, &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;255&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; sensitivity_2, &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;0&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;), (&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;255&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;, &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;255&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;, sensitivity_2))&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;white_3 &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; cv2.inRange(img, (&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;200&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;, &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;200&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;, &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;200&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;), (&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;255&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;, &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;255&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;, &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;255&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;))&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The color threshold is then combined with the saturation thresholded image to generate a merged image which extracts the best of both worlds.&lt;/p&gt;
&lt;p&gt;This code is present in the Thresholder::color_threshold method.
&lt;img src=&quot;https://ssarangi.github.io/img/blog/advanced_lane_finding/merged_thresholding.png&quot; alt=&quot;alt text&quot; title=&quot;Color &amp;#x26; Merged Thresholding with Saturation Thresholding&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; width=&quot;1000&quot; height=&quot;500&quot;&gt;&lt;/p&gt;
&lt;h4 id=&quot;region-of-interest&quot;&gt;Region of Interest&lt;/h4&gt;
&lt;p&gt;During this step we define a region of interest. This is essentially the region which defines the perspective image of the road and the lanes. However we want a top view where the road looks like it is being seen from the top of the road. The image below shows the region of interest and what it is transformed to.
&lt;img src=&quot;https://ssarangi.github.io/img/blog/advanced_lane_finding/roi.png&quot; alt=&quot;alt text&quot; title=&quot;Region of Interest&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; width=&quot;1000&quot; height=&quot;500&quot;&gt;&lt;/p&gt;
&lt;p&gt;With these regions defined we create a perspective transform and we get a transformation matrix and an inverse transformation matrix. The inverse transformation matrix is used at the last step to transform the perspective image back to the original image.&lt;/p&gt;
&lt;p&gt;The code for this part is present in RegionOfInterest::warp_perspective_to_top_down&lt;/p&gt;
&lt;h4 id=&quot;lane-finding-algorithm&quot;&gt;Lane Finding Algorithm&lt;/h4&gt;
&lt;p&gt;The image below shows the perspective image. This image shows the bird’s eye view of the road.
&lt;img src=&quot;https://ssarangi.github.io/img/blog/advanced_lane_finding/perspective.png&quot; alt=&quot;alt text&quot; title=&quot;Perspective Image&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; width=&quot;800&quot; height=&quot;600&quot;&gt;&lt;/p&gt;
&lt;p&gt;The basic idea of the algorithm is very similar to the first project of the Lane finding algorithm. In that project we used Hough transform to find out the lines from the binary images and then use a straight line fitting to generate the lane line. That algorithm didn’t perform very well with curved roads. This algorithm however takes a more intuitive approach to find lane lines.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://ssarangi.github.io/img/blog/advanced_lane_finding/histogram.png&quot; alt=&quot;alt text&quot; title=&quot;Histogram&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; width=&quot;800&quot; height=&quot;600&quot;&gt;&lt;/p&gt;
&lt;p&gt;The first step is to do a histogram on the bottom half of the image to find the high intensities i.e. the white pixels on the image. Once we have the histogram, a window of 100 pixels x 100 pixels is used. The bottom image shows the algorithm as it proceeds. For every window the mean of the pixels is computed and then that is used to search the next window in the pixel image.&lt;/p&gt;
&lt;p&gt;Once this entire process completes, the algorithm figures out the major chunk of pixels. Once we have the left lane pixels and the right lane pixels a polynomial fitting is performed.&lt;/p&gt;
&lt;p&gt;After computing the lanes, we draw them back on the original undistorted image as follows.&lt;/p&gt;
&lt;h5 id=&quot;fine-tuning&quot;&gt;Fine tuning:&lt;/h5&gt;
&lt;ul&gt;
&lt;li&gt;Tried constraining the polyline fit to the center of the box searches but that didn’t work out well.&lt;/li&gt;
&lt;li&gt;Frame averaging - Took about 10 previous frames data to average and get a smooth polyline. This seemed to work decently well for the challenge video.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;This part of the algorithm is present in LaneFinder::full_lane_finding_step&lt;/p&gt;
&lt;h5 id=&quot;short-circuiting-the-lane-finding-algorithm&quot;&gt;Short circuiting the lane finding algorithm&lt;/h5&gt;
&lt;p&gt;Once a full lane finding step is performed for the next frames the search is started around the lanes identified from the previous frame. However, if at any point either the left lane or right lane are not found by this approach a full search is performed.
The partial lane finding code is present in LaneFinder::partial_lane_finding_step
&lt;img src=&quot;https://ssarangi.github.io/img/blog/advanced_lane_finding/lane_finder_algo.png&quot; alt=&quot;alt text&quot; title=&quot;Lane Finding Algorithm&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; width=&quot;800&quot; height=&quot;600&quot;&gt;&lt;/p&gt;
&lt;h4 id=&quot;inverse-transform--final-result&quot;&gt;Inverse Transform &amp;#x26; Final Result&lt;/h4&gt;
&lt;p&gt;&lt;img src=&quot;https://ssarangi.github.io/img/blog/advanced_lane_finding/final_image.png&quot; alt=&quot;alt text&quot; title=&quot;Final Image&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; width=&quot;800&quot; height=&quot;600&quot;&gt;
Once the lanes line region is drawn on the perspective image, the inverse transformation matrix which was obtained when we went from the original image’s region of interest to the bird’s eye view. This gives the final lane finding image.&lt;/p&gt;
&lt;p&gt;The final image overlay and visualization code is also contained in the LaneFinder class in the LaneFinder::overlay methods.&lt;/p&gt;
&lt;h4 id=&quot;computing-the-curvature&quot;&gt;Computing the curvature&lt;/h4&gt;
&lt;p&gt;Computing the curvature involves computing the derivatives of the left lane and the right lane.
&lt;img src=&quot;https://ssarangi.github.io/img/blog/advanced_lane_finding/polynomial.png&quot; alt=&quot;alt text&quot; title=&quot;Polynomial&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; width=&quot;394&quot; height=&quot;80&quot;&gt;
Since we use a 2nd degree polynomial to fit the lines. The reason for using a 2nd degree polynomial was to fit the curves on the road.
&lt;img src=&quot;https://ssarangi.github.io/img/blog/advanced_lane_finding/equation.png&quot; alt=&quot;alt text&quot; title=&quot;Equation of Curvature&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; width=&quot;344&quot; height=&quot;124&quot;&gt;
The above equation is used to compute the curvature.
&lt;img src=&quot;https://ssarangi.github.io/img/blog/advanced_lane_finding/derivatives.png&quot; alt=&quot;alt text&quot; title=&quot;Derivatives&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; width=&quot;406&quot; height=&quot;164&quot;&gt;
The derivative values are the ones shown here and which is obtained from numpy.
LaneFinder::compute_curvature&lt;/p&gt;
&lt;h4 id=&quot;distance-from-center&quot;&gt;Distance from center&lt;/h4&gt;
&lt;p&gt;The equation for computing the distance from the center  is as follows.
((B + C) / 2) - A
where&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;A - Dot product of the center point (height / 2, width) and TO_METER which is the transformation from pixel space to Meters.&lt;/li&gt;
&lt;li&gt;B - First order of derivative of left lane.&lt;/li&gt;
&lt;li&gt;C - First order of derivative of right lane.
LaneFinder::distance_from_center&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id=&quot;problems-with-algorithm&quot;&gt;Problems with Algorithm&lt;/h2&gt;
&lt;h3 id=&quot;simple-project-video&quot;&gt;Simple Project Video:&lt;/h3&gt;
&lt;p&gt;The algorithm tried out in the above sections tend to work well with the project video and didn’t have any kind of project with the partial lane finding steps.&lt;/p&gt;
&lt;h3 id=&quot;challenge-video&quot;&gt;Challenge Video:&lt;/h3&gt;
&lt;p&gt;The biggest problem with the challenge video was the shadows under the bridge where the thresholding seems to detect either way too many regions or not detect the yellow lane lines properly.
&lt;img src=&quot;https://ssarangi.github.io/img/blog/advanced_lane_finding/challenge_frame.png&quot; alt=&quot;alt text&quot; title=&quot;Challenge Frame&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; width=&quot;2560&quot; height=&quot;1440&quot;&gt;
&lt;em&gt;(The output video was missing from the original repository and could not be migrated.)&lt;/em&gt;&lt;/p&gt;
&lt;h3 id=&quot;harder-challenge-video&quot;&gt;Harder Challenge Video:&lt;/h3&gt;
&lt;p&gt;The harder challenge has a greater problem with the core algorithm itself. The problem is that the lane curves very highly. The problem why this becomes problematic is because the right lane curves into the region of the left lane and hence the right lane is mistakenly recognized as a part of the left lane.
&lt;img src=&quot;https://ssarangi.github.io/img/blog/advanced_lane_finding/harder_challenge_frame.png&quot; alt=&quot;alt text&quot; title=&quot;Harder Challenge Frame&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; width=&quot;2560&quot; height=&quot;1440&quot;&gt;
&lt;em&gt;(The output video was missing from the original repository and could not be migrated.)&lt;/em&gt;&lt;/p&gt;
&lt;h2 id=&quot;conclusion&quot;&gt;Conclusion&lt;/h2&gt;
&lt;p&gt;Lane finding is definitely a hard problem which is pretty evident from just the input images provided so considering different lighting conditions and weather conditions as well as road conditions it could be extremely challenging to recognize lane lines accurately. I intend to return to the harder challenges and tweak the techniques to see if I can get a better solution by doing that.&lt;/p&gt;</content:encoded><category>python</category><category>opencv</category></item><item><title>Udacity&apos;s Self Driving Car - Behavior Cloning (P3)</title><link>https://ssarangi.github.io/posts/udacityselfdrivingcar-behaviorcloning/</link><guid isPermaLink="true">https://ssarangi.github.io/posts/udacityselfdrivingcar-behaviorcloning/</guid><description>This post talks about the Behavior Cloning project from Udacity where a car is trained to drive in a simulator.</description><pubDate>Sat, 15 Apr 2017 00:00:00 GMT</pubDate><content:encoded>&lt;h2 id=&quot;behavioral-cloning&quot;&gt;Behavioral Cloning&lt;/h2&gt;
&lt;hr&gt;
&lt;p&gt;After tackling the lane line detection and the traffic sign classification projects from Udacity, the third project to be tackled was another completely different project in terms of what it achieves. The project was centered around the concept of Behavior cloning which meant that we teach a Neural Network model to do a certain task by showing it how it is to be done. The concept is very similar to how you teach babies or even adults in some cases to do certain things.&lt;/p&gt;
&lt;p&gt;The theme of this project was to train a car to drive by itself autonomously. Udacity provided a simulator built in &lt;strong&gt;Unity&lt;/strong&gt; for this purpose. The simulator has two tracks and has two modes it could work on.&lt;/p&gt;
&lt;h3 id=&quot;simulator-modes&quot;&gt;Simulator Modes&lt;/h3&gt;
&lt;ul&gt;
&lt;li&gt;Training Mode: In the training mode the car has to be manually driven and the driving data was stored.&lt;/li&gt;
&lt;li&gt;Autonomous Mode: In the Autonomous mode, a Flask server was run which would use a socket connection to the simulator and then pass in the steering angles and the throttle values to the simulator. The &lt;strong&gt;Steering Angles&lt;/strong&gt; are predicted whereas the throttle values are calculated with a Simple PI Controller.&lt;/li&gt;
&lt;/ul&gt;
&lt;h3 id=&quot;tracks&quot;&gt;Tracks&lt;/h3&gt;
&lt;h4 id=&quot;track-1&quot;&gt;Track 1&lt;/h4&gt;
&lt;p&gt;This track was supposed to be both the training and the testing track. This meant that initially we were supposed to drive the car around on this track like its a game. Once the data is saved for this, this is passed on to a Neural Network to predict the steering angles. The goal was to make the car drive around on this track without going offroad.&lt;/p&gt;
&lt;h5 id=&quot;salient-features&quot;&gt;Salient Features&lt;/h5&gt;
&lt;ul&gt;
&lt;li&gt;Track was &lt;strong&gt;Flat&lt;/strong&gt;&lt;/li&gt;
&lt;li&gt;Favored Left Turns more than Right turns and had a lot of straight road driving.&lt;/li&gt;
&lt;li&gt;Has a Bridge &amp;#x26; Rocks&lt;/li&gt;
&lt;li&gt;Has an evening kind of lighting&lt;/li&gt;
&lt;/ul&gt;
&lt;h4 id=&quot;track-2&quot;&gt;Track 2&lt;/h4&gt;
&lt;p&gt;This was meant to be a challenge track and this was much different than Track 1. This was the challenge track and the idea for this was that once the car was trained on track 1 then how it behaved on Track 2.&lt;/p&gt;
&lt;h5 id=&quot;salient-features-1&quot;&gt;Salient Features&lt;/h5&gt;
&lt;ul&gt;
&lt;li&gt;Track has many up’s and down’s like a roller coaster.&lt;/li&gt;
&lt;li&gt;High speed Driving is much tougher on this than Track 1 because of up’s and down’s.&lt;/li&gt;
&lt;li&gt;Sharp turns present while going up and down.&lt;/li&gt;
&lt;li&gt;Obstacles on side of road are new as compared to Track 1&lt;/li&gt;
&lt;li&gt;Has a center line divide which is absent on Track 1&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id=&quot;data-format-from-training&quot;&gt;Data Format from Training&lt;/h2&gt;
&lt;p&gt;While training the simulator, it asks for a location to save the data. The way the data is stored is like this. There is a &lt;strong&gt;driving_log.csv&lt;/strong&gt; which gets generated and an &lt;strong&gt;IMGS&lt;/strong&gt; folder. The IMGS folder stores the per frame image from the driving. The way the csv is organized is shown below.&lt;/p&gt;
&lt;div class=&quot;table-wrapper&quot; tabindex=&quot;0&quot;&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;center image&lt;/th&gt;
&lt;th&gt;left_image&lt;/th&gt;
&lt;th&gt;right image&lt;/th&gt;
&lt;th&gt;steering_angle&lt;/th&gt;
&lt;th&gt;throttle&lt;/th&gt;
&lt;th&gt;brake&lt;/th&gt;
&lt;th&gt;speed&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;IMG/center_2016_12_01_13_30_48_287.jpg&lt;/td&gt;
&lt;td&gt;IMG/left_2016_12_01_13_30_48_287.jpg&lt;/td&gt;
&lt;td&gt;IMG/right_2016_12_01_3_30_48_287.jpg&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;22.14829&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;
&lt;h3 id=&quot;data-exploration&quot;&gt;Data Exploration&lt;/h3&gt;
&lt;h4 id=&quot;sample-data-from-udacity&quot;&gt;Sample Data from Udacity&lt;/h4&gt;
&lt;p&gt;Udacity provided sample data for people to train their network on. This consisted primarily of driving on Track 1. This is how the data looks when loaded.&lt;/p&gt;
&lt;div class=&quot;table-wrapper&quot; tabindex=&quot;0&quot;&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;steering_angle&lt;/th&gt;
&lt;th&gt;throttle&lt;/th&gt;
&lt;th&gt;brake&lt;/th&gt;
&lt;th&gt;speed&lt;/th&gt;
&lt;th&gt;&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;count&lt;/td&gt;
&lt;td&gt;8036.000000&lt;/td&gt;
&lt;td&gt;8036.000000&lt;/td&gt;
&lt;td&gt;8036.000000&lt;/td&gt;
&lt;td&gt;8036.000000&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;mean&lt;/td&gt;
&lt;td&gt;0.004070&lt;/td&gt;
&lt;td&gt;0.869660&lt;/td&gt;
&lt;td&gt;0.001970&lt;/td&gt;
&lt;td&gt;28.169839&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;std&lt;/td&gt;
&lt;td&gt;0.128840&lt;/td&gt;
&lt;td&gt;0.301326&lt;/td&gt;
&lt;td&gt;0.036565&lt;/td&gt;
&lt;td&gt;6.149327&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;min&lt;/td&gt;
&lt;td&gt;-0.942695&lt;/td&gt;
&lt;td&gt;0.000000&lt;/td&gt;
&lt;td&gt;0.000000&lt;/td&gt;
&lt;td&gt;0.502490&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;25%&lt;/td&gt;
&lt;td&gt;0.000000&lt;/td&gt;
&lt;td&gt;0.985533&lt;/td&gt;
&lt;td&gt;0.000000&lt;/td&gt;
&lt;td&gt;30.183093&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;50%&lt;/td&gt;
&lt;td&gt;0.000000&lt;/td&gt;
&lt;td&gt;0.985533&lt;/td&gt;
&lt;td&gt;0.000000&lt;/td&gt;
&lt;td&gt;30.186400&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;75%&lt;/td&gt;
&lt;td&gt;0.000000&lt;/td&gt;
&lt;td&gt;0.985533&lt;/td&gt;
&lt;td&gt;0.000000&lt;/td&gt;
&lt;td&gt;30.186640&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;max&lt;/td&gt;
&lt;td&gt;1.000000&lt;/td&gt;
&lt;td&gt;0.985533&lt;/td&gt;
&lt;td&gt;1.000000&lt;/td&gt;
&lt;td&gt;30.709360&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;
&lt;p&gt;&lt;img src=&quot;https://ssarangi.github.io/img/blog/behaviorcloning/sample_data_distribution.png&quot; alt=&quot;alt text&quot; title=&quot;Udacity&amp;#x27;s Sample Data Distribution&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; width=&quot;800&quot; height=&quot;600&quot;&gt;&lt;/p&gt;
&lt;h4 id=&quot;training-data-from-manual-training&quot;&gt;Training Data from Manual Training&lt;/h4&gt;
&lt;p&gt;&lt;img src=&quot;https://ssarangi.github.io/img/blog/behaviorcloning/training_data.png&quot; alt=&quot;alt text&quot; title=&quot;Training data&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; width=&quot;1000&quot; height=&quot;1000&quot;&gt;&lt;/p&gt;
&lt;p&gt;The image above shows the training data which is generated from the simulator. All the 3 images from the left, right and center are shown here.&lt;/p&gt;
&lt;h4 id=&quot;dataset-augmentation&quot;&gt;Dataset Augmentation&lt;/h4&gt;
&lt;p&gt;Data Augmentation is a big part of the experiments we carry out with this project. Although,
gathering training data is not very difficult with the simulator, still driving around laps
after laps on the simulator can become cumbersome.&lt;/p&gt;
&lt;p&gt;I realized early on that even though the 0 steering angle had a major predominance in this training data yet throwing it away wasn’t necessary. So the techniques I tried are listed
below.&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Flip the center image &amp;#x26; the steering angle&lt;/li&gt;
&lt;li&gt;Generate about 20 times the data by changing brightness of all center, left &amp;#x26; right images for steering angles less than -0.025 and greater than 0.025.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Since this would have been a lot of data all the augmentation was done while the network was running. I used Pandas DataFrames to generate augmentation by storing columns with the augmentation techniques and during training these were read and the techniques applied appropriately.&lt;/p&gt;
&lt;p&gt;For track 1, I decided against using explicit recovery images. My goal was to get the car to drive and recover without explicitly training on those images. Although this approach worked fine for track 1, this approach didn’t work for track 2. Specifically for track 2 the car had a lot of trouble with the long bars. It would go and hit the road.&lt;/p&gt;
&lt;p&gt;I drove about 5 laps each of track 1 and 5 laps of track 2. Combined with the Sample training images, the augmentation of the images with steering angles either &amp;#x3C; -0.025 or &gt; 0.025 to about 20 times proved to be enough data.&lt;/p&gt;
&lt;h5 id=&quot;recovery-image-for-track-2-where-car-was-having-trouble&quot;&gt;Recovery Image for Track 2 where car was having trouble&lt;/h5&gt;
&lt;p&gt;The car was having trouble in the following location and hence I provided only these specific recovery images. These are mainly for the car to avoid the long poles on track 2.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://ssarangi.github.io/img/blog/behaviorcloning/track2_recovery.gif&quot; alt=&quot;alt text&quot; title=&quot;Track 2 Recovery Img&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; width=&quot;320&quot; height=&quot;160&quot;&gt;&lt;/p&gt;
&lt;h5 id=&quot;using-keras-image-augmentation-generator&quot;&gt;Using Keras Image Augmentation Generator&lt;/h5&gt;
&lt;p&gt;I tried using the Keras Image Augmentation Generator but constantly ran into trouble with it. Keras Image Augmentation generator is a generator which allows about 10 different image augmentation techniques. The one which I really wanted to try out was ZCA whitening. However, for ZCA whitening and std_normalization of the images, the generator had to fit the entire training data which wasn’t possible and keras would constantly run out of memory. So I decided to ditch this method and write my own augmentation methods.&lt;/p&gt;
&lt;p&gt;As such, I wanted to keep the augmentation logic simple to the only 3 augmentation methods applied were as follows:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Changing Brightness Randomly&lt;/li&gt;
&lt;li&gt;Flipping the Center Images for angles != 0&lt;/li&gt;
&lt;li&gt;Adding the Left &amp;#x26; Right Images with a 0.25 degree shift.&lt;/li&gt;
&lt;/ul&gt;
&lt;h5 id=&quot;change-brightness&quot;&gt;Change Brightness&lt;/h5&gt;
&lt;p&gt;&lt;img src=&quot;https://ssarangi.github.io/img/blog/behaviorcloning/change_brightness.png&quot; alt=&quot;alt text&quot; title=&quot;Change Brightness Augmentation&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; width=&quot;2000&quot; height=&quot;500&quot;&gt;&lt;/p&gt;
&lt;h5 id=&quot;flip-image&quot;&gt;Flip Image&lt;/h5&gt;
&lt;p&gt;&lt;img src=&quot;https://ssarangi.github.io/img/blog/behaviorcloning/flipped.png&quot; alt=&quot;alt text&quot; title=&quot;Flipped Image&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; width=&quot;500&quot; height=&quot;800&quot;&gt;&lt;/p&gt;
&lt;h5 id=&quot;track-1---data-distribution&quot;&gt;Track 1 - Data Distribution&lt;/h5&gt;
&lt;p&gt;I drove around 5 laps on this track along the straight path and then took a U-turn to drive around in the reverse direction for 5 more laps.
The data distribution for this data looked like this looked very similar to the sample training data provided since the road architecture is still the same.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://ssarangi.github.io/img/blog/behaviorcloning/track_1_data_distribution.png&quot; alt=&quot;alt text&quot; title=&quot;Track 1 Data Distribution&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; width=&quot;800&quot; height=&quot;600&quot;&gt;&lt;/p&gt;
&lt;h5 id=&quot;track-2---data-distribution&quot;&gt;Track 2 - Data Distribution&lt;/h5&gt;
&lt;p&gt;&lt;img src=&quot;https://ssarangi.github.io/img/blog/behaviorcloning/track_2_data_distribution.png&quot; alt=&quot;alt text&quot; title=&quot;Track 2 Data Distribution&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; width=&quot;800&quot; height=&quot;600&quot;&gt;&lt;/p&gt;
&lt;p&gt;Track 2 has a much better distribution since this road has a lot of turning angles. Also if it noticed, sharp turns are present which makes the training of car to turn sharper angles easier.&lt;/p&gt;
&lt;h5 id=&quot;combined-data-distribution&quot;&gt;Combined Data Distribution&lt;/h5&gt;
&lt;p&gt;Finally I combined all these training data into a single dataset to see how the car performed but before I did that I wanted to visualize what that data looked like so that I get a good idea of what biases it has.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://ssarangi.github.io/img/blog/behaviorcloning/combined_data_distribution.png&quot; alt=&quot;alt text&quot; title=&quot;Combined Data Distribution&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; width=&quot;800&quot; height=&quot;600&quot;&gt;&lt;/p&gt;
&lt;h2 id=&quot;network-selection&quot;&gt;Network Selection&lt;/h2&gt;
&lt;p&gt;After reading a lot of material, I decided to use the Nvidia Model for this behavior training. Comma.ai also has a model which is a good model to be used for this purpose. However, I made a few modifications to the Nvidia model to suit it for this project.
I also used a BatchNormalization layer after the Lambda layers. Along with that I also used L2 regularization on the layers which is not used on the nvidia model. The reason for doing this was because early on I found that since the track was relatively straight on track1 and the scenery was relatively similar, overfitting was happening very soon. It was important to keep it down.&lt;/p&gt;
&lt;h2 id=&quot;network-architecture--parameters&quot;&gt;Network Architecture &amp;#x26; Parameters&lt;/h2&gt;
&lt;p&gt;The nVidia model consists of 9 layers, including a normalization layer, 5 convolutional Layers and 3 fully connected layers. Although the Nvidia model uses YUV planes for the image, I decided to keep it at RGB to keep the problem simple to begin with.&lt;/p&gt;
&lt;p&gt;The convolutional layers in the nvidia model are designed to perform feature extraction and these were designed by nvidia empirically by experimentation by trying out varied layer configurations. The model uses strided convolutions in the first three convolutional layers with 2x2 stride and a 5x5 kernel and a non-strided convolution with a 3x3 kernel size in the last two convolutional layers.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://ssarangi.github.io/img/blog/behaviorcloning/model.png&quot; alt=&quot;alt text&quot; title=&quot;Nvidia Model&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; width=&quot;588&quot; height=&quot;1733&quot;&gt;&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;
&lt;p&gt;&lt;em&gt;Batch Size&lt;/em&gt;: I chose the batch size to be 512. I had a lot of training images so to keep training times a little lower I chose a higher batch size. I also experimented with 256, 128, 64 &amp;#x26; 32 batch sizes and got higher validation loss.&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;&lt;em&gt;Steps Per Epoch&lt;/em&gt;: I kept this to be the size of training data / Batch size.&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;A ‘mean_squared_error’ loss function is used with Adam optimization algorithm configured with a learning rate of 0.0001. This learning rate was selected after bit of trial and error. With larger learning rates (0.1 and 0.001), car was making abrupt steering angle changes, causing bumpy ride.&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;I used the Keras Lambda layers to do Cropping of the image and then the normalization of the image RGB values. This made it easier to not make too much changes to drive.py for doing preprocessing since this was encoded in the model itself. I cropped about 40% from the top since that’s the part which included the sky and trees. However, I think this caused problems on Track 2 when the car was going uphill and some part of the road would get cropped. This is an important point to think about since we can apply cropping reliably in situations where the car is on a flat road.&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;I resized the image to 1/2 its size to (160, 80) from (320, 160) to increase the training size.&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id=&quot;overfitting&quot;&gt;Overfitting&lt;/h2&gt;
&lt;p&gt;One of the biggest problem with this particular project was the overfitting. Within a very few epochs, the network would start overfitting. As I started increased the training data, like combining track 1 with track 2 I noticed that the validation loss kept on increasing. This could be because of the reason that more hidden conv layers might be needed to learn more of the load and the different driving styles.&lt;/p&gt;
&lt;h2 id=&quot;layer-visualization&quot;&gt;Layer Visualization&lt;/h2&gt;
&lt;p&gt;The animated gif’s in this section show the images as the car is driving across the road with the visualization from the last convolution layer from the network before it connects to the Dense Layer. The important thing to notice here are that the images look very similar to the edge detection of the lane lines in this case.&lt;/p&gt;
&lt;h3 id=&quot;track-1-1&quot;&gt;Track 1&lt;/h3&gt;
&lt;p&gt;&lt;img src=&quot;https://ssarangi.github.io/img/blog/behaviorcloning/track1_model_loss.png&quot; alt=&quot;alt text&quot; title=&quot;Track 1 Model Loss&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; width=&quot;800&quot; height=&quot;600&quot;&gt;
&lt;img src=&quot;https://i.imgur.com/BvRziqW.gif&quot; alt=&quot;Track 1 Final Run&quot;&gt;
&lt;img src=&quot;https://i.imgur.com/ynZEdNP.gif&quot; alt=&quot;Track 1 Loss&quot;&gt;&lt;/p&gt;
&lt;p&gt;Track 1 suffers from overfitting within just 5 epochs. This could be due to the use of more training data than necessary. Also, one way to fix this would be to add different kinds of augmentation techniques. However, the car does manage to run nicely on this track and could run continuously across the laps. I tried about 5 laps and it was still running fine.&lt;/p&gt;
&lt;h3 id=&quot;track-2-1&quot;&gt;Track 2&lt;/h3&gt;
&lt;p&gt;&lt;img src=&quot;https://ssarangi.github.io/img/blog/behaviorcloning/track2_model_loss.png&quot; alt=&quot;alt text&quot; title=&quot;Track 1 Model Loss&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; width=&quot;800&quot; height=&quot;600&quot;&gt;
&lt;img src=&quot;https://ssarangi.github.io/img/blog/behaviorcloning/track2_final_run.gif&quot; alt=&quot;alt text&quot; title=&quot;Track 2 Final Run&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; width=&quot;320&quot; height=&quot;160&quot;&gt; &lt;img src=&quot;https://ssarangi.github.io/img/blog/behaviorcloning/track2_conv2d_5_layers.gif&quot; alt=&quot;alt text&quot; title=&quot;Track 2 Conv2d_5&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; width=&quot;500&quot; height=&quot;500&quot;&gt;&lt;/p&gt;
&lt;p&gt;Similar to track 1, track 2 also suffers from the same problem and the same techniques could be applied to track 2. I tried about 2 laps on track 2. The Track 2’s model was built up with the training data from track 1 + track 2. However, the same model didn’t work on track 1 where the car would inevitably go and crash into the wall on the bridge. On Track 2 also while going uphill the car would sometimes go dangerously close to the edge of the road before managing to recover and come back to the center. This could be due to the cropping problem mentioned above.&lt;/p&gt;
&lt;p&gt;Since on track 2 the car’s driving wasn’t very smooth I tried using a Butterworth bandpass filter from scipy but that didn’t give good results. It’s possible that I didn’t choose the right parameters which caused the car to veer of the road when the filter was active.&lt;/p&gt;
&lt;h2 id=&quot;final-results&quot;&gt;Final Results&lt;/h2&gt;
&lt;h3 id=&quot;track-1-2&quot;&gt;Track 1&lt;/h3&gt;
&lt;p&gt;&lt;a href=&quot;https://www.youtube.com/watch?v=-GzA0qTAzWk&quot; title=&quot;Track 1 Run&quot;&gt;&lt;img src=&quot;https://ssarangi.github.io/img/blog/behaviorcloning/track1_youtube.png&quot; alt=&quot;Track 1 Final Result&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; width=&quot;1280&quot; height=&quot;718&quot;&gt;&lt;/a&gt;&lt;/p&gt;
&lt;h3 id=&quot;track-2-2&quot;&gt;Track 2&lt;/h3&gt;
&lt;p&gt;&lt;a href=&quot;https://youtu.be/eooRnceSvno&quot; title=&quot;Track 2 Run&quot;&gt;&lt;img src=&quot;https://ssarangi.github.io/img/blog/behaviorcloning/track2_youtube.png&quot; alt=&quot;Track 2 Final Result&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; width=&quot;857&quot; height=&quot;480&quot;&gt;&lt;/a&gt;&lt;/p&gt;
&lt;h3 id=&quot;future-work&quot;&gt;Future Work:&lt;/h3&gt;
&lt;p&gt;This project is far from being over. Due to time limitations I couldn’t experiment with a custom network as well as the comma.ai network. I definitely want to try out a few things with this.&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;How do the pretrained models perform as compared to the nvidia one we have.&lt;/li&gt;
&lt;li&gt;Try with a limited set of training data as well as reduce the resolution of the training data to make the training faster. This was one of my biggest hurdles since even though I was working out of a machine with a GTI1080 yet my training times were ridiculously high.&lt;/li&gt;
&lt;li&gt;How can we generalize the driving and input data so that the car could perform better on track 2.&lt;/li&gt;
&lt;li&gt;Try more augmentation techniques like moving the image from the center. Also, it would be interesting to see if we could images from the 3rd person camera which is shown in the simulator to be used for training instead of the driver view.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;This project really pushed the “empirical” experiment based approach of deep learning where it was difficult to theorize under what circumstances the car would drive properly and where it wouldn’t. My belief is that if the training time could be reduced, then it was possible to spend a little more time debugging the behavior of the car.&lt;/p&gt;</content:encoded><category>python</category><category>tensorflow</category><category>keras</category></item><item><title>OpenStreetMap rendering and simple Routing.</title><link>https://ssarangi.github.io/posts/openstreetmapsrendering-and-simple-routing/</link><guid isPermaLink="true">https://ssarangi.github.io/posts/openstreetmapsrendering-and-simple-routing/</guid><description>This post is about a small experiment of loading Openstreetmap files and rendering them with both matplotlib and vispy. Also tried a simple routing with dijkstra and A*.</description><pubDate>Fri, 25 Mar 2016 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;&lt;a href=&quot;https://github.com/ssarangi/osmpy&quot;&gt;Github&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;Thanks to &lt;a href=&quot;https://prnbs.github.io&quot;&gt;Pranabesh&lt;/a&gt; for working with me on this.&lt;/p&gt;
&lt;h2 id=&quot;introduction&quot;&gt;Introduction:&lt;/h2&gt;
&lt;p&gt;OpenStreetMaps are becoming more and more popular with open data and truly revolutionizing the maps industry. There are a lot of players in this field but one of the most prominent has been MapBox.
Mapbox has editors to edit and upload OSM files and their rendering is just too beautiful.&lt;/p&gt;
&lt;h2 id=&quot;structure&quot;&gt;Structure:&lt;/h2&gt;
&lt;p&gt;OSM typically uses 2 file formats. XML or PBF. PBF is short for Google’s protocol buffers which is just a compressed format. XML on the other hand is quite verbose.&lt;/p&gt;
&lt;h3 id=&quot;xml-format&quot;&gt;XML format:&lt;/h3&gt;
&lt;p&gt;OSM uses Nodes to represent latitudes and longitudes. These nodes are then combined to form complex structures. For example, a collection of nodes form the buildings or highways or roads etc.
So the structure is pretty simple to understand.&lt;/p&gt;
&lt;p&gt;Each OSM file starts with a Bounding box.&lt;/p&gt;
&lt;pre class=&quot;astro-code astro-code-themes github-light github-dark-dimmed&quot; style=&quot;--shiki-light:#24292e;--shiki-dark:#adbac7;--shiki-light-bg:#fff;--shiki-dark-bg:#22272e; overflow-x: auto;--file-name-offset: -0.75rem;&quot; tabindex=&quot;0&quot; data-language=&quot;xml&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;&amp;#x3C;?&lt;/span&gt;&lt;span style=&quot;--shiki-light:#22863A;--shiki-dark:#8DDB8C&quot;&gt;xml&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; version&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;1.0&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; encoding&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;UTF-8&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;?&gt;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;&amp;#x3C;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#22863A;--shiki-dark:#8DDB8C&quot;&gt;osm&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; version&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;0.6&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; generator&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;CGImap 0.4.0 (23328 thorn-02.openstreetmap.org)&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; copyright&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;OpenStreetMap and contributors&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; attribution&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;http://www.openstreetmap.org/copyright&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; license&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;http://opendatacommons.org/licenses/odbl/1-0/&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;&gt;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;&amp;#x3C;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#22863A;--shiki-dark:#8DDB8C&quot;&gt;bounds&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; minlat&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;37.4081400&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; minlon&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;-121.8716900&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; maxlat&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;37.4099500&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; maxlon&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;-121.8684400&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;/&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The next step are the nodes. Every node has a unique id which is what it is referenced by along with whether its visible on the map or not. The rest of the tags are almost self-explanatory. lat, lon and id are what we will use here.&lt;/p&gt;
&lt;pre class=&quot;astro-code astro-code-themes github-light github-dark-dimmed&quot; style=&quot;--shiki-light:#24292e;--shiki-dark:#adbac7;--shiki-light-bg:#fff;--shiki-dark-bg:#22272e; overflow-x: auto;--file-name-offset: -0.75rem;&quot; tabindex=&quot;0&quot; data-language=&quot;xml&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;&amp;#x3C;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#22863A;--shiki-dark:#8DDB8C&quot;&gt;node&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; id&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;65425741&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; visible&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;true&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; version&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;4&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; changeset&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;3424990&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; timestamp&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;2009-12-22T06:37:55Z&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; user&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;woodpeck_fixbot&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; uid&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;147510&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; lat&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;37.4169957&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; lon&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;-121.8541592&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;/&gt;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;&amp;#x3C;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#22863A;--shiki-dark:#8DDB8C&quot;&gt;node&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; id&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;65443553&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; visible&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;true&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; version&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;6&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; changeset&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;6857224&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; timestamp&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;2011-01-04T03:51:20Z&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; user&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;mk408&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; uid&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;201724&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; lat&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;37.4129151&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; lon&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;-121.8624849&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;/&gt;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;&amp;#x3C;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#22863A;--shiki-dark:#8DDB8C&quot;&gt;node&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; id&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;65452440&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; visible&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;true&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; version&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;6&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; changeset&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;25041843&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; timestamp&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;2014-08-27T00:25:00Z&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; user&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;StellanL&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; uid&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;28775&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; lat&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;37.4100723&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; lon&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;-121.8683465&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;/&gt;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;&amp;#x3C;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#22863A;--shiki-dark:#8DDB8C&quot;&gt;node&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; id&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;65506168&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; visible&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;true&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; version&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;4&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; changeset&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;2817765&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; timestamp&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;2009-10-11T18:55:54Z&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; user&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;woodpeck_fixbot&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; uid&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;147510&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; lat&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;37.4076559&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; lon&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;-121.8691881&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;/&gt;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;&amp;#x3C;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#22863A;--shiki-dark:#8DDB8C&quot;&gt;node&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; id&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;65511824&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; visible&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;true&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; version&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;4&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; changeset&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;3140985&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; timestamp&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;2009-11-17T13:34:37Z&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; user&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;woodpeck_fixbot&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; uid&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;147510&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; lat&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;37.4084658&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; lon&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;-121.8697810&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;/&gt;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;&amp;#x3C;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#22863A;--shiki-dark:#8DDB8C&quot;&gt;node&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; id&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;65588013&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; visible&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;true&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; version&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;7&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; changeset&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;32500664&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; timestamp&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;2015-07-08T17:31:12Z&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; user&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;StellanL&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; uid&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;28775&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; lat&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;37.4091332&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; lon&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;-121.8702756&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;&gt;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;&amp;#x3C;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#22863A;--shiki-dark:#8DDB8C&quot;&gt;tag&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; k&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;highway&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; v&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;traffic_signals&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;/&gt;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;&amp;#x3C;/&lt;/span&gt;&lt;span style=&quot;--shiki-light:#22863A;--shiki-dark:#8DDB8C&quot;&gt;node&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;A way generally represents an object, which could be a highway, residential roads, primary roads, secondary roads etc. A way would use a bunch of node id’s to define its structure. The tag element identifies what the way represents.
It might also contain auxiliary data called Tiger Data which is data released by the US Census Bureau.&lt;/p&gt;
&lt;pre class=&quot;astro-code astro-code-themes github-light github-dark-dimmed&quot; style=&quot;--shiki-light:#24292e;--shiki-dark:#adbac7;--shiki-light-bg:#fff;--shiki-dark-bg:#22272e; overflow-x: auto;--file-name-offset: -0.75rem;&quot; tabindex=&quot;0&quot; data-language=&quot;xml&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;&amp;#x3C;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#22863A;--shiki-dark:#8DDB8C&quot;&gt;way&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; id&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;8940804&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; visible&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;true&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; version&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;8&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; changeset&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;7051091&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; timestamp&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;2011-01-22T14:48:02Z&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; user&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;mk408&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; uid&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;201724&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;&gt;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;  &amp;#x3C;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#22863A;--shiki-dark:#8DDB8C&quot;&gt;nd&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; ref&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;65511824&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;/&gt;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;  &amp;#x3C;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#22863A;--shiki-dark:#8DDB8C&quot;&gt;nd&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; ref&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;65511826&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;/&gt;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;  &amp;#x3C;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#22863A;--shiki-dark:#8DDB8C&quot;&gt;nd&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; ref&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;65511829&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;/&gt;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;  &amp;#x3C;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#22863A;--shiki-dark:#8DDB8C&quot;&gt;nd&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; ref&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;65511832&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;/&gt;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;  &amp;#x3C;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#22863A;--shiki-dark:#8DDB8C&quot;&gt;nd&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; ref&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;65511834&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;/&gt;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;  &amp;#x3C;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#22863A;--shiki-dark:#8DDB8C&quot;&gt;nd&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; ref&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;65511837&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;/&gt;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;  &amp;#x3C;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#22863A;--shiki-dark:#8DDB8C&quot;&gt;nd&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; ref&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;65511838&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;/&gt;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;  &amp;#x3C;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#22863A;--shiki-dark:#8DDB8C&quot;&gt;nd&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; ref&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;313453779&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;/&gt;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;  &amp;#x3C;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#22863A;--shiki-dark:#8DDB8C&quot;&gt;nd&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; ref&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;313453780&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;/&gt;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;  &amp;#x3C;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#22863A;--shiki-dark:#8DDB8C&quot;&gt;nd&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; ref&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;65511840&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;/&gt;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;  &amp;#x3C;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#22863A;--shiki-dark:#8DDB8C&quot;&gt;nd&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; ref&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;65511841&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;/&gt;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;  &amp;#x3C;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#22863A;--shiki-dark:#8DDB8C&quot;&gt;tag&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; k&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;highway&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; v&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;tertiary&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;/&gt;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;  &amp;#x3C;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#22863A;--shiki-dark:#8DDB8C&quot;&gt;tag&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; k&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;name&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; v&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;Junewood Avenue&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;/&gt;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;  &amp;#x3C;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#22863A;--shiki-dark:#8DDB8C&quot;&gt;tag&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; k&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;tiger:cfcc&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; v&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;A41&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;/&gt;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;  &amp;#x3C;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#22863A;--shiki-dark:#8DDB8C&quot;&gt;tag&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; k&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;tiger:county&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; v&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;Santa Clara, CA&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;/&gt;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;  &amp;#x3C;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#22863A;--shiki-dark:#8DDB8C&quot;&gt;tag&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; k&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;tiger:name_base&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; v&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;Junewood&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;/&gt;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;  &amp;#x3C;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#22863A;--shiki-dark:#8DDB8C&quot;&gt;tag&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; k&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;tiger:name_type&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; v&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;Ave&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;/&gt;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;  &amp;#x3C;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#22863A;--shiki-dark:#8DDB8C&quot;&gt;tag&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; k&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;tiger:separated&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; v&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;no&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;/&gt;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;  &amp;#x3C;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#22863A;--shiki-dark:#8DDB8C&quot;&gt;tag&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; k&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;tiger:source&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; v&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;tiger_import_dch_v0.6_20070809&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;/&gt;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;  &amp;#x3C;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#22863A;--shiki-dark:#8DDB8C&quot;&gt;tag&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; k&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;tiger:tlid&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; v&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;122962140:122962213:122962214:122962219:122962228:122962234&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;/&gt;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;  &amp;#x3C;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#22863A;--shiki-dark:#8DDB8C&quot;&gt;tag&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; k&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;tiger:upload_uuid&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; v&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;bulk_upload.pl-9f300d22-5de3-4867-bd5e-8c2a200c22ad&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;/&gt;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;  &amp;#x3C;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#22863A;--shiki-dark:#8DDB8C&quot;&gt;tag&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; k&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;tiger:zip_left&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; v&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;95132&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;/&gt;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;  &amp;#x3C;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#22863A;--shiki-dark:#8DDB8C&quot;&gt;tag&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; k&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;tiger:zip_right&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt; v&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;95132&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;/&gt;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; &amp;#x3C;/&lt;/span&gt;&lt;span style=&quot;--shiki-light:#22863A;--shiki-dark:#8DDB8C&quot;&gt;way&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;With this information we start the process of rendering the maps. A graph is created for the roads which is then used to render.
Used NetworkX to create the graph. Rendering to Matplotlib was done with rules to render different kinds of structures.&lt;/p&gt;
&lt;pre class=&quot;astro-code astro-code-themes github-light github-dark-dimmed&quot; style=&quot;--shiki-light:#24292e;--shiki-dark:#adbac7;--shiki-light-bg:#fff;--shiki-dark-bg:#22272e; overflow-x: auto;--file-name-offset: -0.75rem;&quot; tabindex=&quot;0&quot; data-language=&quot;python&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;class&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#F69D50&quot;&gt; MatplotLibMap&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;:&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;    renderingRules &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; {&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;        &apos;primary&apos;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;: &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;dict&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;                linestyle&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;       =&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt; &apos;-&apos;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;                linewidth&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;       =&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; 6&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;                color&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;           =&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;  (&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;0.933&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;, &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;0.51&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;, &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;0.933&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;),  &lt;/span&gt;&lt;span style=&quot;--shiki-light:#6A737D;--shiki-dark:#768390&quot;&gt;#&apos;#ee82ee&apos;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;                zorder&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;          =&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; 400&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;        ),&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;        &apos;primary_link&apos;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;: &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;dict&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;                linestyle&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;       =&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt; &apos;-&apos;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;                linewidth&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;       =&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; 6&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;                color&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;           =&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; (&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;0.85&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;, &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;0.44&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;, &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;0.84&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;), &lt;/span&gt;&lt;span style=&quot;--shiki-light:#6A737D;--shiki-dark:#768390&quot;&gt;# &apos;#da70d6&apos;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;                zorder&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;          =&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; 300&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;        ),&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;        &apos;secondary&apos;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;: &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;dict&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;                linestyle&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;       =&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt; &apos;-&apos;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;                linewidth&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;       =&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; 6&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;                color&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;           =&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; (&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;0.85&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;, &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;0.75&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;, &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;0.85&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;), &lt;/span&gt;&lt;span style=&quot;--shiki-light:#6A737D;--shiki-dark:#768390&quot;&gt;# &apos;#d8bfd8&apos;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;                zorder&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;          =&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; 200&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;        ),&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;        &apos;secondary_link&apos;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;: &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;dict&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;                linestyle&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;       =&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt; &apos;-&apos;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;                linewidth&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;       =&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; 6&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;                color&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;           =&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; (&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;0.85&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;, &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;0.75&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;, &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;0.85&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;), &lt;/span&gt;&lt;span style=&quot;--shiki-light:#6A737D;--shiki-dark:#768390&quot;&gt;# &apos;#d8bfd8&apos;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;                zorder&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;          =&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; 200&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;        ),&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;        &apos;tertiary&apos;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;: &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;dict&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;                linestyle&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;       =&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt; &apos;-&apos;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;                linewidth&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;       =&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; 4&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;                color&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;           =&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; (&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;1.0&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;, &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;0.0&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;, &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;0.0&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;), &lt;/span&gt;&lt;span style=&quot;--shiki-light:#6A737D;--shiki-dark:#768390&quot;&gt;#(0.0,0.0,0.7),&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;                zorder&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;          =&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; 100&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;        ),&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;        &apos;tertiary_link&apos;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;: &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;dict&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;                linestyle&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;       =&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt; &apos;-&apos;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;                linewidth&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;       =&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; 4&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;                color&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;           =&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; (&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;1.0&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;, &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;0.0&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;, &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;0.0&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;), &lt;/span&gt;&lt;span style=&quot;--shiki-light:#6A737D;--shiki-dark:#768390&quot;&gt;#(0.0,0.0,0.7),&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;                zorder&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;          =&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; 100&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;        ),&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;        &apos;residential&apos;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;: &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;dict&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;                linestyle&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;       =&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt; &apos;-&apos;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;                linewidth&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;       =&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; 1&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;                color&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;           =&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; (&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;1.0&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;, &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;1.0&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;, &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;0.0&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;), &lt;/span&gt;&lt;span style=&quot;--shiki-light:#6A737D;--shiki-dark:#768390&quot;&gt;#(0.1,0.1,0.1),&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;                zorder&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;          =&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; 50&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;        ),&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;        &apos;unclassified&apos;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;: &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;dict&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;                linestyle&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;       =&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt; &apos;:&apos;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;                linewidth&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;       =&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; 1&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;                color&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;           =&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; (&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;0.5&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;0.5&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;0.5&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;),&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;                zorder&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;          =&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; 10&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;        ),&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;        &apos;calculated_path&apos;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;: &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;dict&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;                linestyle&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;       =&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt; &apos;-&apos;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;                linewidth&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;       =&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; 4&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;                color&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;           =&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; (&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;1.0&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;0.0&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;0.0&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;),&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;                zorder&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;          =&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; 2000&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;        ),&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;        &apos;correct_path&apos;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;: &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;dict&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;                linestyle&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;       =&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt; &apos;-&apos;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;                linewidth&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;       =&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; 6&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;                color&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;           =&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; (&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;0.6&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;0.8&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;0.0&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;),&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;                zorder&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;          =&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; 1900&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;        ),&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;        &apos;default&apos;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;: &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;dict&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;                linestyle&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;       =&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt; &apos;-&apos;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;                linewidth&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;       =&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; 3&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;                color&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;           =&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; (&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;1.0&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;, &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;0.48&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;, &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;0.0&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;),&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;                zorder&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;          =&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; 500&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;                ),&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;        &apos;other&apos;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;: &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;dict&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;                linestyle&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;       =&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt; &apos;-&apos;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;                linewidth&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;       =&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; 3&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;                color&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;           =&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; (&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;0.6&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;, &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;0.6&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;, &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;0.6&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;),&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;                zorder&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;          =&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; 500&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;                ),&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;        }&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The render method is also not very complicated.&lt;/p&gt;
&lt;pre class=&quot;astro-code astro-code-themes github-light github-dark-dimmed&quot; style=&quot;--shiki-light:#24292e;--shiki-dark:#adbac7;--shiki-light-bg:#fff;--shiki-dark-bg:#22272e; overflow-x: auto;--file-name-offset: -0.75rem;&quot; tabindex=&quot;0&quot; data-language=&quot;python&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;def&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#DCBDFB&quot;&gt; render&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(self, axes, plot_nodes&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;False&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;):&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;    plt.sca(axes)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;    for&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; idx, nodeID &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;in&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; enumerate&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;self&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;._osm.ways.keys()):&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;        wayTags &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; self&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;._osm.ways[nodeID].tags&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;        wayType &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; None&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;        if&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt; &apos;highway&apos;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt; in&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; wayTags.keys():&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;            wayType &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; wayTags[&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&apos;highway&apos;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;]&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;        if&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; wayType &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;in&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; [&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;                       &apos;primary&apos;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;                       &apos;primary_link&apos;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;                       &apos;unclassified&apos;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;                       &apos;secondary&apos;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;                       &apos;secondary_link&apos;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;                       &apos;tertiary&apos;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;                       &apos;tertiary_link&apos;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;                       &apos;residential&apos;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;                       &apos;trunk&apos;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;                       &apos;trunk_link&apos;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;                       &apos;motorway&apos;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;                       &apos;motorway_link&apos;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;                        ]:&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;            oldX &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; None&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;            oldY &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; None&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;            if&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; wayType &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;in&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; list&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(MatplotLibMap.renderingRules.keys()):&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;                thisRendering &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; MatplotLibMap.renderingRules[wayType]&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;            else&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;:&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;                thisRendering &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; MatplotLibMap.renderingRules[&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&apos;default&apos;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;]&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;            for&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; nCnt, nID &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;in&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; enumerate&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;self&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;._osm.ways[nodeID].nds):&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;                y &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; float&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;self&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;._osm.nodes[nID].lat)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;                x &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; float&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;self&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;._osm.nodes[nID].lon)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;                self&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;._node_map[(x, y)] &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; nID&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;                if&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; oldX &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;is&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; None&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;:&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;                    pass&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;                else&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;:&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;                    plt.plot([oldX,x],[oldY,y],&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;                            marker&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;          =&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt; &apos;&apos;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;                            linestyle&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;       =&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; thisRendering[&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&apos;linestyle&apos;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;],&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;                            linewidth&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;       =&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; thisRendering[&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&apos;linewidth&apos;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;],&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;                            color&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;           =&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; thisRendering[&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&apos;color&apos;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;],&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;                            solid_capstyle&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;  =&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt; &apos;round&apos;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;                            solid_joinstyle&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt; &apos;round&apos;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;                            zorder&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;          =&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; thisRendering[&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&apos;zorder&apos;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;],&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;                            picker&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;2&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;                    )&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;                    if&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; plot_nodes &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;==&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; True&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt; and&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; (nCnt &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;==&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; 0&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt; or&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; nCnt &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;==&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; len&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;self&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;._osm.ways[nodeID].nds) &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;-&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; 1&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;):&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;                        plt.plot(x, y,&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&apos;ro&apos;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;, &lt;/span&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;zorder&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;5&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;                oldX &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; x&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;                oldY &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; y&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;    self&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;._fig.canvas.mpl_connect(&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&apos;pick_event&apos;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;, &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;self&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;.__onclick__)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;    plt.draw()&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;h3 id=&quot;matplotlib-rendering&quot;&gt;Matplotlib Rendering&lt;/h3&gt;
&lt;p&gt;&lt;img src=&quot;https://raw.githubusercontent.com/ssarangi/osmpy/master/shortest_path.png&quot; alt=&quot;Matplotlib&quot;&gt;&lt;/p&gt;
&lt;h3 id=&quot;vispy-rendering&quot;&gt;Vispy Rendering&lt;/h3&gt;
&lt;p&gt;&lt;img src=&quot;https://raw.githubusercontent.com/ssarangi/osmpy/master/vispy_rendering.png&quot; alt=&quot;Matplotlib&quot;&gt;&lt;/p&gt;</content:encoded><category>python</category><category>openstreetmap</category><category>vispy</category><category>dijkstra</category><category>a-star</category></item><item><title>Incremental text search using tries and ternary trees</title><link>https://ssarangi.github.io/posts/incremental-search/</link><guid isPermaLink="true">https://ssarangi.github.io/posts/incremental-search/</guid><description>While brushing up on algorithms, I decided to reimplement tries and ternary trees and in the process thought it would be a good idea to write a UI accompanying the code so its easier to see. Could further add timing code to this to see the performance too.</description><pubDate>Thu, 04 Feb 2016 00:00:00 GMT</pubDate><content:encoded>&lt;h2 id=&quot;introduction&quot;&gt;Introduction:&lt;/h2&gt;
&lt;p&gt;According to Wikipedia, tries can be defined as “an ordered tree data structure that is used to store a dynamic set or associative array where the keys are usually strings.” Now the definition sounds complex so I prefer to always use examples to illustrate what things would mean. For now, from a very high level its a tree structure whose nodes are used to store strings and the typical use of such datastructures are in string lookup / lexical sorting / doing incremental search etc.&lt;/p&gt;
&lt;p&gt;In this post, I implemented tries and ternary trees to demonstrate the differences between the two datastructures and how they are created and traversed.&lt;/p&gt;
&lt;p&gt;Since I am too lazy to draw the images myself, I have shamelessly stole the images from other sites. All credits to the original authors.&lt;/p&gt;
&lt;h2 id=&quot;ternary-trees&quot;&gt;Ternary Trees:&lt;/h2&gt;
&lt;p&gt;&lt;img src=&quot;https://courses.teresco.org/cs211_f09/labs/lexicon/trie2.jpg&quot; alt=&quot;Nested If Else Block&quot; title=&quot;Tries&quot;&gt;&lt;/p&gt;
&lt;p&gt;The image above shows how a ternary tree would look. So every node in this tree essentially has lets say 26 or 52 (if we distinguish between lowercase and uppercase characters) children. Obviously things can be optimized to eliminate nodes which would never appear but for simplicity lets say we have that many child nodes.&lt;/p&gt;
&lt;p&gt;Now lets say we have created this tree from a dictionary of words. The words which are there in the dictionary are
as follows.&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;are&lt;/li&gt;
&lt;li&gt;as&lt;/li&gt;
&lt;li&gt;dot&lt;/li&gt;
&lt;li&gt;news&lt;/li&gt;
&lt;li&gt;not&lt;/li&gt;
&lt;li&gt;zen&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The above ternary tree shows how the datastructure would look when its created from the words above. So question is how would incremental search work with such a datastructure. So let’s say, someone presses ‘n’ on the keyboard.&lt;/p&gt;
&lt;p&gt;So the root node is first looked up to see if it has any child with ‘n’. In this case it does find it. So then it does a Depth First Search from the node ‘n’ and it keeps on accumulating the child nodes (or the characters the child nodes represent). So in this case it could go to ‘e’ or ‘o’ respectively and so on and so forth.&lt;/p&gt;
&lt;p&gt;So the 2 words it gets are “news” and “not”.&lt;/p&gt;
&lt;p&gt;Now the user presses ‘e’. So, then ‘e’ is searched on all child nodes of ‘n’. Once the ‘e’ node is found again a DFS is done and the result is “ews”. So the user is only returned “news” and no other result.&lt;/p&gt;
&lt;h3 id=&quot;optimizing-the-search&quot;&gt;Optimizing the search:&lt;/h3&gt;
&lt;p&gt;Obviously doing DFS at every node can quickly become slow and painful and hence we could &lt;b&gt;cache&lt;/b&gt; the results of any previous dfs performed at that node and look at up. If it’s available then that can be returned directly.&lt;/p&gt;
&lt;h2 id=&quot;tries&quot;&gt;Tries:&lt;/h2&gt;
&lt;p&gt;Although the exact classification of tries has been confusing around the web, for the purposes of this post lets assume that its a compressed ternary tree (Obviously do read up the actual definitions to be clear).&lt;/p&gt;
&lt;p&gt;So lets look at a few cases from the image above. “d” —&gt; “o” —&gt; “t” have no child nodes in between. So we could safely collapse these to one node “dot”. The same goes for “ews” and “ot”. So the difference between this and the previous tree is that they it uses far fewer nodes and hence the dfs is faster, specially for longer words. It also reduces the memory requirements for storing this in memory.&lt;/p&gt;
&lt;h2 id=&quot;implementation&quot;&gt;Implementation:&lt;/h2&gt;
&lt;p&gt;I have added UI which can compute the trees and store them as python pickle files. That way it doesn’t have to be recreated everytime and it can be loaded from disk. The code is far from perfect but does demonstrate the idea.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://ssarangi.github.io/img/blog/incremental_search/incremental_search.gif&quot; alt=&quot;UI&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; width=&quot;369&quot; height=&quot;343&quot;&gt;&lt;/p&gt;</content:encoded><category>tries</category><category>ternary trees</category><category>python</category><category>pyside</category></item><item><title>Adapting Luajit for Python 3</title><link>https://ssarangi.github.io/posts/luajit-for-python/</link><guid isPermaLink="true">https://ssarangi.github.io/posts/luajit-for-python/</guid><description>I have looked at spidermonkey and a few other JIT engines and kept on reading about LuaJit over and over again. I finally decided to look at Luajit closely and realized that it is a pretty complex piece of engineering but done single-handedly by Mike Pall. The number of source files is way smaller compared to spidermonkey or v8 but that&apos;s probably because Lua is much simpler than javascript. Since is fairly closer to python I wanted to try my hands out to modify Luajit&apos;s code to experiment if I could run basic python on it.</description><pubDate>Thu, 04 Feb 2016 00:00:00 GMT</pubDate><content:encoded>&lt;h2 id=&quot;luajit-is-for-lua-&quot;&gt;Luajit is for Lua !!!&lt;/h2&gt;
&lt;p&gt;I know… But out of curiosity and wanting to see more of it I decided to go ahead and try and see what I could get Luajit to run.&lt;/p&gt;
&lt;h2 id=&quot;luajit-architecture&quot;&gt;Luajit Architecture:&lt;/h2&gt;
&lt;p&gt;Luajit is written completely in C and has a huge helping hand from assembly too. But overall if you look at it, it has the same number of things a regular jitter does.&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;It lexes and parses lua code&lt;/li&gt;
&lt;li&gt;Emit’s a bytecode from it.&lt;/li&gt;
&lt;li&gt;From the bytecode it emits an SSA IR&lt;/li&gt;
&lt;li&gt;It then optimizes the SSA IR.&lt;/li&gt;
&lt;li&gt;It also has an interpreter which acts as a tracing jit.&lt;/li&gt;
&lt;li&gt;Finally after identifying hotspots, it runs it on the full jit.&lt;/li&gt;
&lt;li&gt;Garbage collection is also present and Mike had ideas on the next gen garbage collectors too.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id=&quot;information&quot;&gt;Information:&lt;/h2&gt;
&lt;p&gt;Look on the Luajit Wiki. It has tons of information.&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href=&quot;https://news.ycombinator.com/item?id=6819702&quot;&gt;Luajit trace compiler&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href=&quot;http://blog.reverberate.org/2012/12/hello-jit-world-joy-of-simple-jits.html&quot;&gt;Luajit Dynasm&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href=&quot;http://stackoverflow.com/questions/7167566/luajit-2-optimization-guide&quot;&gt;Optimization links&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href=&quot;https://pwparchive.wordpress.com/2012/10/16/peeking-inside-luajit/&quot;&gt;Peeking inside Luajit&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id=&quot;code-which-is-working&quot;&gt;Code which is working:&lt;/h2&gt;
&lt;p&gt;&lt;a href=&quot;https://github.com/ssarangi/luajit&quot;&gt;Github Link&lt;/a&gt;&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Simple functions&lt;/li&gt;
&lt;li&gt;lists&lt;/li&gt;
&lt;li&gt;While loops&lt;/li&gt;
&lt;li&gt;If-else-elseif control flow&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;There is a hell lot of work left and am not even sure if everything can be / should be supported by this. If you would like to collaborate on this, drop me a line.&lt;/p&gt;
&lt;pre class=&quot;astro-code astro-code-themes github-light github-dark-dimmed&quot; style=&quot;--shiki-light:#24292e;--shiki-dark:#adbac7;--shiki-light-bg:#fff;--shiki-dark-bg:#22272e; overflow-x: auto;--file-name-offset: -0.75rem;&quot; tabindex=&quot;0&quot; data-language=&quot;python&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;def&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#DCBDFB&quot;&gt; foo&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;():&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;    print&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;hello world&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;    a &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; 3&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;    b &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; 2&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;    c &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; a &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;*&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; b&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;    return&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; c&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;g &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; foo()&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;print&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(g)&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;pre class=&quot;astro-code astro-code-themes github-light github-dark-dimmed&quot; style=&quot;--shiki-light:#24292e;--shiki-dark:#adbac7;--shiki-light-bg:#fff;--shiki-dark-bg:#22272e; overflow-x: auto;--file-name-offset: -0.75rem;&quot; tabindex=&quot;0&quot; data-language=&quot;python&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;def&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#DCBDFB&quot;&gt; foo&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;():&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;    a &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; 3&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;    b &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; 3&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;    if&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; a &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;&gt;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; b:&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;        print&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;a &gt; b&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;    elif&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; a &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;==&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; b:&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;        print&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;a == b&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;    else&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;:&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;        print&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;b &gt; a&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;foo()&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;pre class=&quot;astro-code astro-code-themes github-light github-dark-dimmed&quot; style=&quot;--shiki-light:#24292e;--shiki-dark:#adbac7;--shiki-light-bg:#fff;--shiki-dark-bg:#22272e; overflow-x: auto;--file-name-offset: -0.75rem;&quot; tabindex=&quot;0&quot; data-language=&quot;python&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;days &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; [&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;Sunday&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;, &lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;Monday&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;, &lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;Tuesday&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;, &lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;Wednesday&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;, &lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;Thursday&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;, &lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;Friday&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;, &lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;Saturday&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;]&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;print&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(days[&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;0&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;])&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;i &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; 3&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;i &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; i &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;+&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; 1&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;print&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(days[i])&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;print&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(days[&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;4&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;])&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;pre class=&quot;astro-code astro-code-themes github-light github-dark-dimmed&quot; style=&quot;--shiki-light:#24292e;--shiki-dark:#adbac7;--shiki-light-bg:#fff;--shiki-dark-bg:#22272e; overflow-x: auto;--file-name-offset: -0.75rem;&quot; tabindex=&quot;0&quot; data-language=&quot;python&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;def&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#DCBDFB&quot;&gt; foo&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;():&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;    a &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; 1&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;    while&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; a &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;&amp;#x3C;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; 10&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;:&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;        print&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(a)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;        a &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; a &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;+&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; 1&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;foo()&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;</content:encoded><category>jit</category><category>lua</category><category>luajit</category><category>python</category></item><item><title>Loopy - A Structurizer for generating structured control flow code.</title><link>https://ssarangi.github.io/posts/loopy-structurizer/</link><guid isPermaLink="true">https://ssarangi.github.io/posts/loopy-structurizer/</guid><description>A structurizer is used when the IR is unstructured but the code to be generated needs to be structured. A structured control flow represents idioms which are generally present in higher-level languages such as if-else, for-loop, while-loop, do-while etc.</description><pubDate>Wed, 20 Jan 2016 00:00:00 GMT</pubDate><content:encoded>&lt;h2 id=&quot;introduction&quot;&gt;Introduction&lt;/h2&gt;
&lt;p&gt;&lt;img src=&quot;https://ssarangi.github.io/img/blog/spiderjit/nested_if_cfg.png&quot; alt=&quot;Nested If Else Block&quot; title=&quot;Nested If-else CFG&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; width=&quot;254&quot; height=&quot;614&quot;&gt;&lt;/p&gt;
&lt;p&gt;In this post I will explain an algorithm which I call Loopy. I developed it independently while looking for algorithms for structuring and is similar to Relooper (used in Emscripten) and Hammock graph based algorithms. However, my intention with this post is to simplify the algorithm and not speak of jargons generally associated with describing such algorithms.&lt;/p&gt;
&lt;h2 id=&quot;some-definitions&quot;&gt;Some Definitions:&lt;/h2&gt;
&lt;p&gt;Before I jump into the algorithm, I will just give a brief explanation of the two terms about control flow graphs.&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;
&lt;p&gt;Structured Control Flow: Control Flow graph which uses if-else, for-loop, while-loop and do-while in the intermediate representation is called structured control flow.&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;Unstructured Control Flow: Look at the picture above. It’s the basic blocks linked to one another using branch instructions. There is no structure really. However, when we do if-else or our favorite loops (if-else, for, while, do-while), there is a structure since there are no branches which can go haywire.&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id=&quot;algorithm&quot;&gt;Algorithm:&lt;/h2&gt;
&lt;p&gt;Before I proceed, I describe a couple of classes which I have defined for creating the structurizer. There are 2 classes which will be used for representing the structure of the Control Flow.&lt;/p&gt;
&lt;pre class=&quot;astro-code astro-code-themes github-light github-dark-dimmed&quot; style=&quot;--shiki-light:#24292e;--shiki-dark:#adbac7;--shiki-light-bg:#fff;--shiki-dark-bg:#22272e; overflow-x: auto;--file-name-offset: -0.75rem;&quot; tabindex=&quot;0&quot; data-language=&quot;python&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;class&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#F69D50&quot;&gt; Root&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;:&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;    def&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; __init__&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(self, parent):&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;        self&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;.processed &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; False&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;        self&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;.__parent &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; parent&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;        if&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; parent &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;is&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt; not&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; None&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;:&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;            parent.next &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; self&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#DCBDFB&quot;&gt;    @&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;property&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;    def&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#DCBDFB&quot;&gt; parent&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(self):&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;        return&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; self&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;.__parent&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#DCBDFB&quot;&gt;    @parent.setter&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;    def&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#DCBDFB&quot;&gt; parent&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(self, p):&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;        self&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;.__parent &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; p&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;        if&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; p &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;is&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt; not&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; None&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;:&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;            p.next &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; self&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;class&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#F69D50&quot;&gt; ControlFlowBlock&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt;Root&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;):&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;    def&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; __init__&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(self, parent &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; None&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;):&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;        Root.&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;__init__&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;self&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;, parent)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;        self&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;.true_block &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; None&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;        self&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;.false_block &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; None&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;        self&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;.cmp_inst &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; None&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;        self&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;.nested &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; None&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;    def&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; __str__&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(self):&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;        s &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt; &quot;CFB: True: &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#F47067&quot;&gt;%s&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt; &amp;#x3C;--&gt; False: &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#F47067&quot;&gt;%s&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt; %&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; (&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;self&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;.true_block, &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;self&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;.false_block)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;        return&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; s&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;class&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#F69D50&quot;&gt; Nested&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt;Root&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;):&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;    def&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; __init__&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(self, bb, parent&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;None&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;, next&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;None&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;):&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;        Root.&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;__init__&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;self&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;, parent)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;        self&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;.bb &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; bb&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;        self&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;.next &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; next&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;    def&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; __str__&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(self):&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;        s &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt; &quot;Nested: &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#F47067&quot;&gt;%s&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt; %&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; self&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;.bb&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;        return&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; s&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The Nested class is used to store a nested block. The idea is that nested blocks link to each other like a linked list, have a link to their parent and create a nesting kind of structure.&lt;/p&gt;
&lt;p&gt;Each Control Flow Block is used to recreate the structure and use the true &amp;#x26; false blocks to represent the control flow structure.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://ssarangi.github.io/img/blog/spiderjit/nested_if_structurizer.png&quot; alt=&quot;Nested If Else Structurized diagram&quot; title=&quot;Nested If Structurized&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; width=&quot;1357&quot; height=&quot;577&quot;&gt;&lt;/p&gt;
&lt;p&gt;The interesting blocks here are the CFB’s (Control Flow Block’s). The CFB’s encompass all the true blocks and false blocks. So the code generation becomes simpler since now the control flow block with keep all the control flow information within it.&lt;/p&gt;
&lt;p&gt;The algorithm is simple.&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Create Nested Blocks for every basic block&lt;/li&gt;
&lt;li&gt;Create a Control Flow Block for every block which has a Conditional Branch.&lt;/li&gt;
&lt;li&gt;Do a breadth first search starting with the function’s entry block.&lt;/li&gt;
&lt;li&gt;Associate the nested blocks for the true block and false block if a basic block has a conditional branch terminator.&lt;/li&gt;
&lt;li&gt;If the terminator is a branch terminator then just add it to it’s parent list.&lt;/li&gt;
&lt;li&gt;If we visit a block which has been visited before during the bfs, we have hit a convergence point.&lt;/li&gt;
&lt;li&gt;Once we get a convergence point, go to the nearest ancestor which is a Control Flow Block and which has not been PROCESSED.&lt;/li&gt;
&lt;li&gt;Set the next block for the CFB as the convergence point.&lt;/li&gt;
&lt;li&gt;Set the CFB as been PROCESSED.&lt;/li&gt;
&lt;/ul&gt;</content:encoded><category>compiler</category><category>structurizer</category><category>unstructured control flow</category><category>structured control flow</category><category>cfg</category></item><item><title>Constant Propagation &amp; Folding.</title><link>https://ssarangi.github.io/posts/constant-propagation-and-folding/</link><guid isPermaLink="true">https://ssarangi.github.io/posts/constant-propagation-and-folding/</guid><description>Constant propagation and folding are generic optimizations which can greatly simplify the intermediate representation. It is the basis for other optimizations like loop unrolling, loop simplification, function inlining etc.</description><pubDate>Sun, 17 Jan 2016 00:00:00 GMT</pubDate><content:encoded>&lt;h2 id=&quot;constant-folding&quot;&gt;Constant Folding:&lt;/h2&gt;
&lt;p&gt;Before we define constant propagation, let’s define constant folding. Constant folding is a step in which you can “fold” instructions when their operands are constants.
In the instruction below we don’t have to emit instructions for computing the result of this instruction. This instruction has both the operands as constants and hence can be computed at compile time, thus getting the result &lt;code&gt;a = 20&lt;/code&gt;.&lt;/p&gt;
&lt;pre class=&quot;astro-code astro-code-themes github-light github-dark-dimmed&quot; style=&quot;--shiki-light:#24292e;--shiki-dark:#adbac7;--shiki-light-bg:#fff;--shiki-dark-bg:#22272e; overflow-x: auto;--file-name-offset: -0.75rem;&quot; tabindex=&quot;0&quot; data-language=&quot;python&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;a &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; 5&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt; *&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; 4&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;h3 id=&quot;constant-propagation&quot;&gt;Constant Propagation:&lt;/h3&gt;
&lt;p&gt;Constant propagation on the other hand means what the name suggests. It propagates the constants. But propagate where ? Again lets look at an example.&lt;/p&gt;
&lt;pre class=&quot;astro-code astro-code-themes github-light github-dark-dimmed&quot; style=&quot;--shiki-light:#24292e;--shiki-dark:#adbac7;--shiki-light-bg:#fff;--shiki-dark-bg:#22272e; overflow-x: auto;--file-name-offset: -0.75rem;&quot; tabindex=&quot;0&quot; data-language=&quot;python&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;def&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#DCBDFB&quot;&gt; foo&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;():&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;  a &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; 5&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt; *&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; 4&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;  b &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; 3&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt; *&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; a&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;  c &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; 9&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt; *&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; b&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;  d &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; 10&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt; *&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; c&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;  return&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; d&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;def&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#DCBDFB&quot;&gt; main&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;():&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;  g &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; foo()&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;  return&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; g &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;*&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; 10&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;So this is a contrived example but it does demonstrate the idea very well. In the function “foo”, ‘a’ can be constant folded to 20.
Now, if we propagate ‘a’ to the next instruction &lt;code&gt;b = 3 * a&lt;/code&gt;, which means at compile time we replace the value of ‘a’ with the constant 20. Now again, &lt;code&gt;b = 3 * 20&lt;/code&gt;, so ‘b’ becomes 60. Now, similarly ‘c’ becomes 540 and then d becomes 5400.&lt;/p&gt;
&lt;p&gt;Cool, so now all the code can be replaced with an equivalent instruction. So the simplified instruction sequence would look like this.&lt;/p&gt;
&lt;pre class=&quot;astro-code astro-code-themes github-light github-dark-dimmed&quot; style=&quot;--shiki-light:#24292e;--shiki-dark:#adbac7;--shiki-light-bg:#fff;--shiki-dark-bg:#22272e; overflow-x: auto;--file-name-offset: -0.75rem;&quot; tabindex=&quot;0&quot; data-language=&quot;python&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;def&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#DCBDFB&quot;&gt; foo&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;():&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;  return&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; 5400&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;def&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#DCBDFB&quot;&gt; main&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;():&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;  g &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; foo()&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;  return&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; g &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;*&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; 10&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;So are we done yet? Not really. As you can see that the function foo() will always return 5400. So do we really need to generate code for executing a function? No. We can simplify this function further be replacing its caller by it’s value.
So &lt;code&gt;g = 5400&lt;/code&gt; and then ‘main’ can constant fold the next instruction to like this.&lt;/p&gt;
&lt;pre class=&quot;astro-code astro-code-themes github-light github-dark-dimmed&quot; style=&quot;--shiki-light:#24292e;--shiki-dark:#adbac7;--shiki-light-bg:#fff;--shiki-dark-bg:#22272e; overflow-x: auto;--file-name-offset: -0.75rem;&quot; tabindex=&quot;0&quot; data-language=&quot;python&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;def&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#DCBDFB&quot;&gt; main&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;():&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;  return&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; 54000&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Now this could be further optimized, if it’s the main function in python to return 54000 to the stdout. So this demonstrates the idea of Constant Propagation.&lt;/p&gt;
&lt;p&gt;The implementation algorithm is also very intuitive as it does the same as the process we described above.&lt;/p&gt;
&lt;p&gt;In my project, I am doing constant folding as the IR builder is building the instructions themselves making it very simple and also when I do constant propagation since that reveals opportunities for folding again.&lt;/p&gt;
&lt;p&gt;Look at the files, const_progation.py &amp;#x26; irbuilder.py for seeing the idea.&lt;/p&gt;</content:encoded><category>compiler</category><category>constant propagation</category><category>constant folding</category></item><item><title>Dominator Trees - Implementation and Why are they needed?</title><link>https://ssarangi.github.io/posts/dominator-trees/</link><guid isPermaLink="true">https://ssarangi.github.io/posts/dominator-trees/</guid><description>A Dominator Tree is a CFG analysis which is used in compilers to aid in other optimizations. This is useful when doing optimizations like code-hoisting or code sinking etc.</description><pubDate>Sun, 17 Jan 2016 00:00:00 GMT</pubDate><content:encoded>&lt;h2 id=&quot;so-what-are-dominator-trees&quot;&gt;So what are Dominator Trees:&lt;/h2&gt;
&lt;p&gt;Before understanding Dominator Trees, the more fundamental concept to understand is Dominance. According to Wikipedia, Dominance is defined as a node’s property where a node ‘d’ is said to dominate node ‘n’ if every path from the entry node to the node ‘n’ has to pass through node ‘d’. Every node always dominates itself.&lt;/p&gt;
&lt;p&gt;So what is the tree then? Well, when this property is computed for all the nodes in a graph then it is called a Dominator Tree.&lt;/p&gt;
&lt;pre class=&quot;astro-code astro-code-themes github-light github-dark-dimmed&quot; style=&quot;--shiki-light:#24292e;--shiki-dark:#adbac7;--shiki-light-bg:#fff;--shiki-dark-bg:#22272e; overflow-x: auto;--file-name-offset: -0.75rem;&quot; tabindex=&quot;0&quot; data-language=&quot;python&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;def&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#DCBDFB&quot;&gt; main&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(a, b, c):&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;    d &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; 0&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;    if&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; a &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;&gt;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; b:&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;        if&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; b &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;&gt;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; c:&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;            c &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; b&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;            a &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; b&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;            d &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; a &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;*&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; c&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;        else&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;:&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;            c &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; a &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;*&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; b&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;            a &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; c &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;-&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; b&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;            d &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; a &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;-&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; c&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;        d &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; d &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;*&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; 5&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;    else&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;:&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;        d &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; b&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;    return&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; d&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;So the next question which comes is what can be considered as a node here in this graph. We just have python code you might say. Look at the IR below. This IR contains instructions and basic blocks.&lt;/p&gt;
&lt;p&gt;So what’s a Basic Block? It is essentially a set of instructions which has some kind of terminator instruction at it’s end (for example, branch instructions, or return instructions) etc.&lt;/p&gt;
&lt;p&gt;So each instruction can be considered a node or each basic block can be considered a node.&lt;/p&gt;
&lt;pre class=&quot;astro-code astro-code-themes github-light github-dark-dimmed&quot; style=&quot;--shiki-light:#24292e;--shiki-dark:#adbac7;--shiki-light-bg:#fff;--shiki-dark-bg:#22272e; overflow-x: auto;--file-name-offset: -0.75rem;&quot; tabindex=&quot;0&quot; data-language=&quot;plaintext&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span&gt;Module: root_module&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;define  main(a, b, c) {&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;&amp;#x3C;entry&gt;:&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;    %0 = icmp sle a, b&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;    br %0 1, label %then, label %else&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;&amp;#x3C;then&gt;: ; pred: [&apos;entry&apos;]&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;    %1 = icmp sle b, c&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;    br %1 1, label %then_1, label %else_1&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;&amp;#x3C;then_1&gt;: ; pred: [&apos;then&apos;]&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;    %mult = mul(b, b)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;    br endif_1&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;&amp;#x3C;else_1&gt;: ; pred: [&apos;then&apos;]&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;    %mult1 = mul(b, b)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;    %sub = sub(%mult1, b)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;    %sub1 = sub(%sub, %mult1)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;    br endif_1&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;&amp;#x3C;endif_1&gt;: ; pred: [&apos;else_1&apos;, &apos;then_1&apos;]&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;    %mult2 = mul(%sub1, 5)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;    br endif&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;&amp;#x3C;else&gt;: ; pred: [&apos;entry&apos;]&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;    br endif&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;&amp;#x3C;endif&gt;: ; pred: [&apos;else&apos;, &apos;endif_1&apos;]&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;    return b&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;}&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;&lt;img src=&quot;https://ssarangi.github.io/img/blog/spiderjit/nested_if_cfg.png&quot; alt=&quot;CFG&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; width=&quot;254&quot; height=&quot;614&quot;&gt;&lt;/p&gt;
&lt;p&gt;Now look at the CFG above. From this diagram, it is easy to say which node dominates which node or in other words which Basic Block dominates which other basic block.&lt;/p&gt;
&lt;p&gt;For example, entry dominates all other basic blocks but it is not dominated by any other basic block.&lt;/p&gt;
&lt;p&gt;“else_1” is dominated by the entry and then since to get to else_1 we have to go through else_1.&lt;/p&gt;
&lt;p&gt;Interesting block is the “endif_1”. This block is only dominated by “entry” and “then” but not by “else_1” and “then_1”. The reason for this is we can come from “entry” -&gt; “then” -&gt; “else_1” -&gt; “endif_1” or we can take the path
“entry” -&gt; “then” -&gt; “then_1” -&gt; “endif_1”.&lt;/p&gt;
&lt;p&gt;So in both the paths, only “entry” and “then” are common so we consider only those blocks.&lt;/p&gt;
&lt;p&gt;The output below shows the dominance relation between all the blocks. This datastructure is called a Dominance Tree.&lt;/p&gt;
&lt;pre class=&quot;astro-code astro-code-themes github-light github-dark-dimmed&quot; style=&quot;--shiki-light:#24292e;--shiki-dark:#adbac7;--shiki-light-bg:#fff;--shiki-dark-bg:#22272e; overflow-x: auto;--file-name-offset: -0.75rem;&quot; tabindex=&quot;0&quot; data-language=&quot;plaintext&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span&gt;&amp;#x3C;else&gt; --&gt; [&amp;#x3C;else&gt;, &amp;#x3C;entry&gt;]&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;&amp;#x3C;endif_1&gt; --&gt; [&amp;#x3C;endif_1&gt;, &amp;#x3C;entry&gt;, &amp;#x3C;then&gt;]&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;&amp;#x3C;endif&gt; --&gt; [&amp;#x3C;entry&gt;, &amp;#x3C;endif&gt;]&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;&amp;#x3C;then_1&gt; --&gt; [&amp;#x3C;then_1&gt;, &amp;#x3C;entry&gt;, &amp;#x3C;then&gt;]&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;&amp;#x3C;entry&gt; --&gt; [&amp;#x3C;entry&gt;]&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;&amp;#x3C;else_1&gt; --&gt; [&amp;#x3C;entry&gt;, &amp;#x3C;else_1&gt;, &amp;#x3C;then&gt;]&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;&amp;#x3C;then&gt; --&gt; [&amp;#x3C;entry&gt;, &amp;#x3C;then&gt;]&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;h2 id=&quot;implementation&quot;&gt;Implementation:&lt;/h2&gt;
&lt;p&gt;Wikipedia lists a very simple algorithm for computing dominance. However, there are quite a few good algorithms with discussions on their running times which should be read in order to fully understand the implications of this algorithm.&lt;/p&gt;
&lt;pre class=&quot;astro-code astro-code-themes github-light github-dark-dimmed&quot; style=&quot;--shiki-light:#24292e;--shiki-dark:#adbac7;--shiki-light-bg:#fff;--shiki-dark-bg:#22272e; overflow-x: auto;--file-name-offset: -0.75rem;&quot; tabindex=&quot;0&quot; data-language=&quot;plaintext&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span&gt;// dominator of the start node is the start itself&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;Dom(n0) = {n0}&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;// for all other nodes, set all nodes as the dominators&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;for each n in N - {n0}&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;    Dom(n) = N;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;// iteratively eliminate nodes that are not dominators&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;while changes in any Dom(n)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;    for each n in N - {n0}:&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;        Dom(n) = {n} union with intersection over Dom(p) for all p in pred(n)&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The idea is simple. First mark the Dominator block of the entry block as itself since we cannot reach the entry block in any other way.&lt;/p&gt;
&lt;p&gt;Next iterate through all the blocks (other than the entry block) and set their dominators to all the blocks (except for the entry block)&lt;/p&gt;
&lt;p&gt;Now as long as we have some changes in the dominator tree blocks, for each block other than entry, do the following.&lt;/p&gt;
&lt;p&gt;Take the dominators for all the predecessor blocks and intersect them and then union the block itself with the result. So why does this work. Lets take the example of “endif_1”.&lt;/p&gt;
&lt;p&gt;So assuming all the other blocks have already computed their dominators, lets trace this through.&lt;/p&gt;
&lt;p&gt;Predecessor’s for “endif_1” are “else_1” and “then_1”.
Dom(else_1) = { “entry”, “then”, “else_1” }
Dom(then_1) = { “entry”, “then”, “then_1” }&lt;/p&gt;
&lt;p&gt;so Dom(else_1) intersect Dom(then_1) = { “entry”, “then” }&lt;/p&gt;
&lt;p&gt;So Dom(endif_1) = {endif_1} U { “entry”, “then” }
So Dom(endif_1) = { “entry”, “then”, “endif_1” }&lt;/p&gt;
&lt;p&gt;For my implementation, I have used python set’s to do the intersect method which makes it very simple to implement.&lt;/p&gt;
&lt;h2 id=&quot;usage&quot;&gt;Usage:&lt;/h2&gt;
&lt;p&gt;Dominator Tree’s are used in a variety of optimizations. We can use it for doing loop optimizations, reordering basic blocks, scheduling instructions, structuring control flow (which was my main interest in implementing this).&lt;/p&gt;
&lt;p&gt;Future post I will touch upon how to use this information in the structurizer.&lt;/p&gt;</content:encoded><category>compiler</category><category>dominator trees</category><category>post dominator tree</category></item><item><title>Compiler IR - Considerations and Implementation</title><link>https://ssarangi.github.io/posts/compiler-ir-considerations-and-implementation/</link><guid isPermaLink="true">https://ssarangi.github.io/posts/compiler-ir-considerations-and-implementation/</guid><description>A Compiler intermediate representation - consideration and implementation details.</description><pubDate>Sat, 16 Jan 2016 00:00:00 GMT</pubDate><content:encoded>&lt;h2 id=&quot;compiler-ir&quot;&gt;Compiler IR:&lt;/h2&gt;
&lt;p&gt;While working on my project for a jit in python, one of the main design considerations was to select an appropriate IR. An IR or Intermediate Representation of an input source program. So what’s the use of an IR ? Why the hell do I need one to begin with?&lt;/p&gt;
&lt;p&gt;So let’s say you decided to write a compiler for some esoteric language you devised or even python / ruby for that matter. Let’s say your goal is to generate javascript code out of it. So you implemented a lexer and parser for the language, or better still, if you are writing a compiler for python, you used the module “ast” to build an Abstract Syntax Tree for it and then you can directly do a source to source translation to javascript from the ast itself. Infact, PythonJS(&lt;a href=&quot;https://github.com/PythonJS/PythonJS&quot;&gt;https://github.com/PythonJS/PythonJS&lt;/a&gt;) does exactly this.&lt;/p&gt;
&lt;p&gt;So you might be wondering at this point, what else is needed? Where does the whole IR concept fit into this?&lt;/p&gt;
&lt;h2 id=&quot;why-an-ir&quot;&gt;Why an IR?&lt;/h2&gt;
&lt;p&gt;So let’s continue the hypothetical situation we described above. You generated javascript code but realized it is not optimal. So you write some optimizations on the AST. All fine and dandy indeed !!!&lt;/p&gt;
&lt;p&gt;Now you decide to write a compiler for another language but you find out that you need another AST and for generating efficient code you need to rewrite all your optimizations. So this is where an Intermediate Representation (IR) is useful.&lt;/p&gt;
&lt;p&gt;An IR is just a mid-level language to which your compiler frontends convert to. An IR could be either very close to the original source language or be very close to the target representation and all these depend on your requirements.&lt;/p&gt;
&lt;p&gt;For example, LLVM has 2 levels or IR. A regular IR which is relatively high level and supports a lot of constructs like switch, functions, modules etc yet lower level than C / C++. The other level of IR is called MachineIR which is very close to current hardware targets like x86, SPARC, Mips etc. LLVM does a lot of optimizations at it’s higher level IR before translating to a lower level IR.&lt;/p&gt;
&lt;h2 id=&quot;ir---design-considerations&quot;&gt;IR - Design Considerations:&lt;/h2&gt;
&lt;p&gt;So what are the things to consider when designing an IR. An IR changes the direction in which one has to write optimizations and hence poses a quite an important decision.&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;SSA vs Non-SSA&lt;/li&gt;
&lt;li&gt;Unstructured vs Structured Control Flow&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;These are the 2 main considerations I can think of while designing an IR. The Dragon book for example prescribes a TAC (Three Address Code) format of representing IR. The IR looks very low level and very close to x86 assembly. However, it is Non-SSA and uses a Structured Control Flow approach.&lt;/p&gt;
&lt;p&gt;I am more familiar with SSA based IR from LLVM which also happens to be Unstructured Control Flow.&lt;/p&gt;
&lt;p&gt;So what is the difference between Structured and Unstructured control Flow?&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Structured Control flow uses the higher level language constructs like if-then-else-endif, while, do-while, for etc.&lt;/li&gt;
&lt;li&gt;Unstructured on the other hand can only use branch instructions, conditional branch instructions (goto kind of instructions) and labels to implement the higher level control flow.&lt;/li&gt;
&lt;li&gt;Typed or Typeless&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;So some of the advantages of SSA are that def-use chains are easy to build since every instruction is a def and all values are unique. Also a variable cannot have more than one def making it easy to do optimizations like Dead Code Elimination.&lt;/p&gt;
&lt;p&gt;However, on the other hand, it introduces the concept of phi-nodes which means there have to be optimizations which convert to a phi-based ir and then during the code generation phase have to convert it out of a phi-based ir to the target code. These are not relatively straightforward to do.&lt;/p&gt;
&lt;p&gt;Now with unstructured control flow, its easy to build the control flow graph and do analysis passes like DominatorTree pass etc but it makes it slightly more complicated to do optimizations on loops (since loops have to be recreated, induction variables recalculated etc) as well as generating code for a structured control flow target. For example, from python to javascript, if we go through an unstructued control flow based IR then we have to write a structurizer which will get back the original structure from the IR to generate code since javascript doesn’t support goto’s and this means that fairly complex and inefficient code will be generated in this case. Needing a structurizer also means that in some cases it might be very difficult to recreate the structure if the mid-end optimizations are not careful.&lt;/p&gt;
&lt;p&gt;Typed or Typeless discussions are important based on the language from which it is being converted. For example for languages which are statically typed (like C/C++/C#,Java etc) it makes sense to use a typed IR. However, for dynamic languages like python, ruby etc using a typed IR has its own challenges. You need some kind of type inference to try and predict the types or you need to create complex objects which would need boxing and unboxing when working with different types.&lt;/p&gt;
&lt;h2 id=&quot;ir-implementation&quot;&gt;IR Implementation:&lt;/h2&gt;
&lt;p&gt;So I had worked with LLVM before and was fairly familiar with it’s IR when I started working on this project. So my first obvious choice would be have been to integrate LLVM using some python bindings. However, most of the python bindings seem to be have died since people couldn’t keep up with maintaining with llvm’s pace (or maybe they got bored ;) ). The only stable version which seems to be there is llvmlite from numba project. Looking at llvmlite, I decided to try and write my own IR which would be very similar to llvm ir but would not require me to link to llvm libraries.&lt;/p&gt;
&lt;p&gt;The code is not very complex for designing the IR, but for the first step, I decided to make it typeless so that I don’t have to write a type inferer immediately. This made it relatively easier to do the translation from the frontend to the midend. However, it does make doing optimizations more difficult, specially alias analysis etc. However, I think I can live without that for now.&lt;/p&gt;
&lt;pre class=&quot;astro-code astro-code-themes github-light github-dark-dimmed&quot; style=&quot;--shiki-light:#24292e;--shiki-dark:#adbac7;--shiki-light-bg:#fff;--shiki-dark-bg:#22272e; overflow-x: auto;--file-name-offset: -0.75rem;&quot; tabindex=&quot;0&quot; data-language=&quot;python&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;def&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#DCBDFB&quot;&gt; main&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(a, b, c):&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;    d &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; 0&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;    if&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; a &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;&gt;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; b:&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;        if&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; b &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;&gt;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; c:&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;            c &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; b&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;            a &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; b&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;            d &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; a &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;*&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; c&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;        else&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;:&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;            c &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; a &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;*&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; b&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;            a &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; c &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;-&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; b&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;            d &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; a &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;-&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; c&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;        d &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; d &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;*&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; 5&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;    else&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;:&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;        d &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; b&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;    return&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; d&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;For this python source above, my current IR being generated is shown below.&lt;/p&gt;
&lt;pre class=&quot;astro-code astro-code-themes github-light github-dark-dimmed&quot; style=&quot;--shiki-light:#24292e;--shiki-dark:#adbac7;--shiki-light-bg:#fff;--shiki-dark-bg:#22272e; overflow-x: auto;--file-name-offset: -0.75rem;&quot; tabindex=&quot;0&quot; data-language=&quot;plaintext&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span&gt;Module: root_module&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;define  main(a, b, c) {&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;&amp;#x3C;entry&gt;:&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;    %0 = icmp sle a, b&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;    br %0 1, label %then, label %else&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;&amp;#x3C;then&gt;: ; pred: [&apos;entry&apos;]&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;    %1 = icmp sle b, c&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;    br %1 1, label %then_1, label %else_1&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;&amp;#x3C;then_1&gt;: ; pred: [&apos;then&apos;]&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;    %mult = mul(b, b)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;    br endif_1&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;&amp;#x3C;else_1&gt;: ; pred: [&apos;then&apos;]&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;    %mult1 = mul(b, b)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;    %sub = sub(%mult1, b)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;    %sub1 = sub(%sub, %mult1)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;    br endif_1&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;&amp;#x3C;endif_1&gt;: ; pred: [&apos;else_1&apos;, &apos;then_1&apos;]&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;    %mult2 = mul(%sub1, 5)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;    br endif&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;&amp;#x3C;else&gt;: ; pred: [&apos;entry&apos;]&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;    br endif&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;&amp;#x3C;endif&gt;: ; pred: [&apos;else&apos;, &apos;endif_1&apos;]&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;    return b&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;}&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The IR looks very similar to llvm (except for its printing aspects). The output below shows how the control flow graph looks for this particular IR.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://ssarangi.github.io/img/blog/spiderjit/nested_if_cfg.png&quot; alt=&quot;CFG&quot; title=&quot;Control Flow Graph&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; width=&quot;254&quot; height=&quot;614&quot;&gt;&lt;/p&gt;
&lt;p&gt;In future posts, I will talk more about some of the optimizations I am doing with this IR.&lt;/p&gt;
&lt;p&gt;&lt;a href=&quot;https://github.com/ssarangi/spiderjit&quot;&gt;Github Link&lt;/a&gt;&lt;/p&gt;</content:encoded><category>compiler</category><category>intermediate representation</category><category>llvm ir</category></item><item><title>Debugger for PyVyM</title><link>https://ssarangi.github.io/posts/debugger-for-pyvym/</link><guid isPermaLink="true">https://ssarangi.github.io/posts/debugger-for-pyvym/</guid><description>PyVyM is a Virtual Machine for python and this post describes a debugger for that VM.</description><pubDate>Mon, 11 Jan 2016 00:00:00 GMT</pubDate><content:encoded>&lt;h2 id=&quot;introduction&quot;&gt;Introduction:&lt;/h2&gt;
&lt;p&gt;While writing the PyVyM virtual machine, I decided to write a debugger. This might not be the most sophisticated debugger but it does demonstrate the ideas needed to write this.&lt;/p&gt;
&lt;p&gt;gdb, lldb are examples of more complicated debuggers used for debugging executable code. However, for the purposes of our VM, its pretty simple to write a debugger.&lt;/p&gt;
&lt;h2 id=&quot;soul-of-the-debugger&quot;&gt;Soul of the debugger:&lt;/h2&gt;
&lt;p&gt;The soul of the debugger is the ability of the debugger to cause the underlying machine to stop whenever it hits it’s breakpoints. At that point it should be able to introspect the various states of the machine. For example, registers, memory, threads running etc etc.&lt;/p&gt;
&lt;p&gt;However, our VM is a single threaded VM. Very simple indeed. So the main thing needed for this debugger is the ability to set breakpoints and stop at those. It took me a couple of hours to get this to work and I was decently pleased with it considering how simple it was.&lt;/p&gt;
&lt;h2 id=&quot;abilities-of-the-debugger&quot;&gt;Abilities of the debugger:&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;next - Execute Next Instruction&lt;/li&gt;
&lt;li&gt;run - Run VM&lt;/li&gt;
&lt;li&gt;set bp &lt;loc&gt; - Set Breakpoint at loc&lt;/loc&gt;&lt;/li&gt;
&lt;li&gt;disable bp &lt;loc&gt; - Disable Breakpoint at loc&lt;/loc&gt;&lt;/li&gt;
&lt;li&gt;clear bp &lt;loc&gt; - Disable Breakpoint at loc&lt;/loc&gt;&lt;/li&gt;
&lt;li&gt;clear all bps - Clear all Breakpoints&lt;/li&gt;
&lt;li&gt;view source &lt;loc&gt; - View Source. If no loc is specified entire source is shown&lt;/loc&gt;&lt;/li&gt;
&lt;li&gt;view locals - View the Local variables&lt;/li&gt;
&lt;li&gt;view globals - View the Global variables&lt;/li&gt;
&lt;li&gt;view local &lt;var&gt; - View local var&lt;/var&gt;&lt;/li&gt;
&lt;li&gt;view global &lt;var&gt; - View global var&lt;/var&gt;&lt;/li&gt;
&lt;li&gt;view backtrace - View the BackTrace&lt;/li&gt;
&lt;li&gt;view bp - View Breakpoints&lt;/li&gt;
&lt;li&gt;help - Display this help&lt;/li&gt;
&lt;li&gt;quit - Quit&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id=&quot;implementation&quot;&gt;Implementation:&lt;/h2&gt;
&lt;p&gt;The main logic of the debugger is as follows:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Ability to translate byte code offsets to line numbers&lt;/li&gt;
&lt;li&gt;Setup a list of breakpoints (each breakpoint is a line number).&lt;/li&gt;
&lt;li&gt;Run an execute loop which would execute each instruction and then check if any of the breakpoint has been hit.&lt;/li&gt;
&lt;li&gt;Since one python line source could correspond to multiple lines of the bytecode, so executing the &lt;b&gt;next&lt;/b&gt; instruction should be able to execute all lines of the bytecode corresponding to that line.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id=&quot;line-number-mapping&quot;&gt;Line Number mapping:&lt;/h2&gt;
&lt;p&gt;Line number mapping is the of the two main things needed to implement this. The file debugger_support.py contains a class called LineNo which translates this.&lt;/p&gt;
&lt;p&gt;Python code object contains a field called co_lnotab which contains the mapping of the bytecode range and the source line number.&lt;/p&gt;
&lt;pre class=&quot;astro-code astro-code-themes github-light github-dark-dimmed&quot; style=&quot;--shiki-light:#24292e;--shiki-dark:#adbac7;--shiki-light-bg:#fff;--shiki-dark-bg:#22272e; overflow-x: auto;--file-name-offset: -0.75rem;&quot; tabindex=&quot;0&quot; data-language=&quot;python&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;class&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#F69D50&quot;&gt; LineNo&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;:&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;    &quot;&quot;&quot;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;    Described in http://svn.python.org/projects/python/trunk/Objects/lnotab_notes.txt&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;    &quot;&quot;&quot;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;    def&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; __init__&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(self, start_lineno, co_lnotab, source, filename):&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;        self&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;.__start_lineno &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; start_lineno&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;        self&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;.__byte_increments &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; list&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(co_lnotab[&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;0&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;::&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;2&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;])&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;        self&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;.__line_increments &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; list&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(co_lnotab[&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;1&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;::&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;2&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;])&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;        self&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;.__source &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; source&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;        self&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;.__filename &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; filename&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;        self&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;.__currently_executed_line &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; None&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;    def&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#DCBDFB&quot;&gt; line_number&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(self, ip):&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;        lineno &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; addr &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; 0&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;        for&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; addr_incr, line_incr &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;in&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; zip&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;self&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;.__byte_increments, &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;self&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;.__line_increments):&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;            addr &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;+=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; addr_incr&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;            if&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; addr &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;&gt;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; ip:&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;                return&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; lineno &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;+&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; self&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;.__start_lineno&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;            lineno &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;+=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; line_incr&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;        return&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; lineno &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;+&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; self&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;.__start_lineno&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;This function described above gives the line numbers. The byte code offsets are not absolute but rather in increments. The link here (&lt;a href=&quot;http://svn.python.org/projects/python/trunk/Objects/lnotab_notes.txt&quot;&gt;http://svn.python.org/projects/python/trunk/Objects/lnotab_notes.txt&lt;/a&gt;) gives the details of how the code is presented.&lt;/p&gt;
&lt;p&gt;So we keep on adding increments to see if we have exceeded the ip. Once the address exceeds the current ip we get the next line number mapped to the bytecode.&lt;/p&gt;
&lt;h2 id=&quot;overriding-the-execute-loop&quot;&gt;Overriding the execute loop:&lt;/h2&gt;
&lt;p&gt;The VM has an execute loop which runs through the code. So the debugger has to make sure that it runs its own execute loop. The VM exposes interfaces to run an individual instruction so it’s easy for the debugger to run each instruction and then check all the breakpoints set.&lt;/p&gt;
&lt;p&gt;However, problems arise when function calls are made. Function calls are typically implemented with its own stack frame and once a new stack frame is created, the call function calls the vm’s execute method. So let’s see why this is a problem.&lt;/p&gt;
&lt;pre class=&quot;astro-code astro-code-themes github-light github-dark-dimmed&quot; style=&quot;--shiki-light:#24292e;--shiki-dark:#adbac7;--shiki-light-bg:#fff;--shiki-dark-bg:#22272e; overflow-x: auto;--file-name-offset: -0.75rem;&quot; tabindex=&quot;0&quot; data-language=&quot;python&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;def&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#DCBDFB&quot;&gt; foo&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(a, b):&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;  return&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; a &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;+&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; b&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;foo(&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;3&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;, &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;4&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;)&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;So the debugger starts with setting up the module, loading the arguments into the stack. Till this point it is fine since the debugger is running its own execute method which checks for breakpoints.&lt;/p&gt;
&lt;p&gt;Now once it sees the function call foo(3, 4), it has to invoke the function foo’s code. At this point, the VM calls its own execute method which is a loop to run all the instructions in the foo method. This means that the debugger has no way of checking after every instruction if any breakpoint has been set.&lt;/p&gt;
&lt;h2 id=&quot;debugger-hookup&quot;&gt;Debugger Hookup:&lt;/h2&gt;
&lt;p&gt;So the way to hook up the debugger to the vm is for the debugger to override the vm’s execute method. This is very simple. When we instantiate the vm’s object we just override the execute method with the debugger’s execute method.&lt;/p&gt;
&lt;pre class=&quot;astro-code astro-code-themes github-light github-dark-dimmed&quot; style=&quot;--shiki-light:#24292e;--shiki-dark:#adbac7;--shiki-light-bg:#fff;--shiki-dark-bg:#22272e; overflow-x: auto;--file-name-offset: -0.75rem;&quot; tabindex=&quot;0&quot; data-language=&quot;python&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;def&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#DCBDFB&quot;&gt; initialize_vm&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(self, code, source, filename):&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;    self&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;.__vm &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; BytecodeVM(code, source, filename)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;    config &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; VMConfig()&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;    self&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;.__vm.config &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; config&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;    config.show_disassembly &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; True&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;    self&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;.__vm.execute &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; self&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;.execute&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;This snippet of code shows how the execute method is overridden. The last line does it. Once this is done, the only problem remaining is that the debugger’s execute method has a loop which displays a prompt.&lt;/p&gt;
&lt;pre class=&quot;astro-code astro-code-themes github-light github-dark-dimmed&quot; style=&quot;--shiki-light:#24292e;--shiki-dark:#adbac7;--shiki-light-bg:#fff;--shiki-dark-bg:#22272e; overflow-x: auto;--file-name-offset: -0.75rem;&quot; tabindex=&quot;0&quot; data-language=&quot;python&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;def&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#DCBDFB&quot;&gt; execute&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(self, call_from_vm &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; True&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;):&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;    while&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; True&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;:&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;        arg1 &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; None&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;        if&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt; not&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; call_from_vm:&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;            cmd_res &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; self&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;.display_prompt()&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;            if&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; isinstance&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(cmd_res, &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;tuple&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;):&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;                cmd &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; cmd_res[&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;0&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;]&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;                arg1 &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; cmd_res[&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;1&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;]&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;            else&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;:&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;                cmd &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; cmd_res&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;        else&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;:&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;            cmd &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; DebuggerCmds.&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;VM_RUN&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;This snippet shows how the prompt is to be displayed. So when the VM CALL_FUNCTION calls the execute method it comes into the debugger’s execute method with the “call_from_vm” parameter “True”. This indicates that this call is to be run continuously without displaying the prompt till we hit a breakpoint.&lt;/p&gt;
&lt;h2 id=&quot;viewing--setting-locals&quot;&gt;Viewing &amp;#x26; Setting Locals:&lt;/h2&gt;
&lt;p&gt;Viewing and setting locals is similar to the how the VM internally does it. It goes up the exec_frame stack to find out if any of the exec_frames contains the local we are interested in. If so it set’s it or returns it. The only thing to take into account here is that the type of the variable is unknown and the value to set from the command prompt comes in the form of a string. So the solution to this is to check for the existing type of the variable. If we can typecast the set value to that type then we continue otherwise we set it as a string itself.&lt;/p&gt;
&lt;h2 id=&quot;debugger-in-action&quot;&gt;Debugger in action:&lt;/h2&gt;
&lt;p&gt;&lt;img src=&quot;https://ssarangi.github.io/img/blog/pyvym/debugger.gif&quot; alt=&quot;Debugger&quot; title=&quot;Debugger&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; width=&quot;908&quot; height=&quot;502&quot;&gt;&lt;/p&gt;</content:encoded><category>python3</category><category>debugger</category><category>virtualmachine</category><category>vm</category></item><item><title>A simple python based JIT</title><link>https://ssarangi.github.io/posts/simple-python-based-jitter/</link><guid isPermaLink="true">https://ssarangi.github.io/posts/simple-python-based-jitter/</guid><description>A simple python based JIT with a C module backend which uses mmap to execute arbitrary x86 code.</description><pubDate>Sun, 10 Jan 2016 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;#Introduction:
JIT or Just-in-Time compilation has been a hot topic in recent years with Jitter’s popping up for all kinds of languages. The most prominent one of them is the one for Javascript from both Google and Mozilla known as V8 and SpiderMonkey respectively. Obviously, python followers joined the bandwagon with a bunch of jitter’s for python, like pyston from Dropbox, Numba from the Anaconda distribution folks etc.&lt;/p&gt;
&lt;p&gt;Jit-ting is fundamental idea in which the compiler generates the x86 assembly code at runtime and executes it. Optimizations like hot-spot analysis etc are done to further decide when to actually JIT and this field is huge and growing.&lt;/p&gt;
&lt;p&gt;So this weekend, I was tinkering around with some simple jit blogs when I came across a very interesting blog which is a must read. &lt;a href=&quot;http://blog.reverberate.org/2012/12/hello-jit-world-joy-of-simple-jits.html&quot;&gt;Joy of Simple JIT&lt;/a&gt;
As I was looking at the code I decided to see if I can make it run via python so that I could write a simple jit for my own language some day. The blog well explains the concepts relating to using mmap with PROT_EXEC permissions so execute assembly. Read the blog before you continue any further.&lt;/p&gt;
&lt;h2 id=&quot;first-step&quot;&gt;First Step&lt;/h2&gt;
&lt;p&gt;For this to work, the first step is to write a C python module which would expose the mmap functionality to python. Python gives access to mmap features from its internal libraries but they don’t give the EXEC permissions and hence I had to go the C route. So the first step was to look up how a C python module would work. For now, this code works only on linux, since it is a pain to get python modules compiling in windows.&lt;/p&gt;
&lt;p&gt;So create a new directory which will hold the project. In that create a file named &lt;b&gt;setup.py&lt;/b&gt;.&lt;/p&gt;
&lt;pre class=&quot;astro-code astro-code-themes github-light github-dark-dimmed&quot; style=&quot;--shiki-light:#24292e;--shiki-dark:#adbac7;--shiki-light-bg:#fff;--shiki-dark-bg:#22272e; overflow-x: auto;--file-name-offset: -0.75rem;&quot; tabindex=&quot;0&quot; data-language=&quot;python&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;from&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; distutils.core &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;import&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; setup, Extension&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;jittermodule &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; Extension(&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&apos;jitter&apos;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;                         sources&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; [&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&apos;jitter/jittermodule.c&apos;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;])&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;setup (&lt;/span&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;name&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt; &apos;PackageName&apos;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;       version&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt; &apos;1.0&apos;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;       description&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt; &apos;This is the Jitter Module for Pyasm2&apos;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;       ext_modules&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; [jittermodule])&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The first line is the important line. It shows the location of the jitter module. According to this setup.py, my C file is located in a folder named &lt;b&gt;jitter&lt;/b&gt;&lt;/p&gt;
&lt;p&gt;Name this file &lt;b&gt;jittermodule.c&lt;/b&gt;&lt;/p&gt;
&lt;pre class=&quot;astro-code astro-code-themes github-light github-dark-dimmed&quot; style=&quot;--shiki-light:#24292e;--shiki-dark:#adbac7;--shiki-light-bg:#fff;--shiki-dark-bg:#22272e; overflow-x: auto;--file-name-offset: -0.75rem;&quot; tabindex=&quot;0&quot; data-language=&quot;c&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;#include&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt; &amp;#x3C;Python.h&gt;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;#include&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt; &amp;#x3C;stdio.h&gt;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;#include&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt; &amp;#x3C;stdlib.h&gt;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;#include&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt; &amp;#x3C;string.h&gt;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;#include&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt; &amp;#x3C;sys/mman.h&gt;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#6A737D;--shiki-dark:#768390&quot;&gt;// Allocates RW memory of given size and returns a pointer to it. On failure,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#6A737D;--shiki-dark:#768390&quot;&gt;// prints out the error and returns NULL. mmap is used to allocate, so&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#6A737D;--shiki-dark:#768390&quot;&gt;// deallocation has to be done with munmap, and the memory is allocated&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#6A737D;--shiki-dark:#768390&quot;&gt;// on a page boundary so it&apos;s suitable for calling mprotect.&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;void*&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#DCBDFB&quot;&gt; alloc_writable_memory&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;size_t&lt;/span&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt; size&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;) {&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;  void*&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; ptr &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#DCBDFB&quot;&gt; mmap&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;0&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;, size,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;                   PROT_READ &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;|&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; PROT_WRITE,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;                   MAP_PRIVATE &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;|&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; MAP_ANON, &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;-&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;1&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;, &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;0&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;);&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;  if&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; (ptr &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;==&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; (&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;void*&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;)&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;-&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;1&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;) {&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#DCBDFB&quot;&gt;    perror&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;mmap&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;);&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;    return&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; NULL&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;  }&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;  return&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; ptr;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;}&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#6A737D;--shiki-dark:#768390&quot;&gt;// Sets a RX permission on the given memory, which must be page-aligned. Returns&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#6A737D;--shiki-dark:#768390&quot;&gt;// 0 on success. On failure, prints out the error and returns -1.&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;int&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#DCBDFB&quot;&gt; make_memory_executable&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;void*&lt;/span&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt; m&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;, &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;size_t&lt;/span&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt; size&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;) {&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;  if&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; (&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#DCBDFB&quot;&gt;mprotect&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(m, size, PROT_READ &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;|&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; PROT_EXEC) &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;==&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;1&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;) {&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#DCBDFB&quot;&gt;    perror&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;mprotect&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;);&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;    return&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;1&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;  }&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;  return&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; 0&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;}&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;void&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#DCBDFB&quot;&gt; emit_code_into_memory&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;unsigned&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt; char*&lt;/span&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt; m&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;, &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;unsigned&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt; char&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt; *&lt;/span&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;code&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;) {&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#6A737D;--shiki-dark:#768390&quot;&gt;  /*&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#6A737D;--shiki-dark:#768390&quot;&gt;  unsigned char code[] = {&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#6A737D;--shiki-dark:#768390&quot;&gt;    0x48, 0x89, 0xf8,                   // mov %rdi, %rax&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#6A737D;--shiki-dark:#768390&quot;&gt;    0x48, 0x83, 0xc0, 0x04,             // add $4, %rax&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#6A737D;--shiki-dark:#768390&quot;&gt;    0xc3                                // ret&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#6A737D;--shiki-dark:#768390&quot;&gt;  };&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#6A737D;--shiki-dark:#768390&quot;&gt;  */&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#DCBDFB&quot;&gt;  memcpy&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(m, code, &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;sizeof&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(code));&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;}&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;const&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt; size_t&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; SIZE &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; 1024&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;typedef&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt; long&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; (&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;*&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;JittedFunc)(&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;long&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;);&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#6A737D;--shiki-dark:#768390&quot;&gt;// Allocates RW memory, emits the code into it and sets it to RX before&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#6A737D;--shiki-dark:#768390&quot;&gt;// executing.&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;int&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#DCBDFB&quot;&gt; emit_to_rw_run_from_rx&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;unsigned&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt; char&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt; *&lt;/span&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;code&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;) {&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;  void*&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; m &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#DCBDFB&quot;&gt; alloc_writable_memory&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(SIZE);&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#DCBDFB&quot;&gt;  emit_code_into_memory&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(m, code);&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#DCBDFB&quot;&gt;  make_memory_executable&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(m, SIZE);&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;  JittedFunc func &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; m;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;  int&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; result &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#DCBDFB&quot;&gt; func&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;4&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;);&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;  return&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; result;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;}&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;static&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; PyObject &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;*&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;JitterError;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;static&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; PyObject&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;*&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#DCBDFB&quot;&gt; jitter_jit&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(PyObject &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;*&lt;/span&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;self&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;, PyObject &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;*&lt;/span&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;args&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;) {&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;    const&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt; char*&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; str;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;    char&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt; *&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; buf;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;    Py_ssize_t&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; count;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;    PyObject &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;*&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; result;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;    int&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; i;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;    if&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; (&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;!&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#DCBDFB&quot;&gt;PyArg_ParseTuple&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(args, &lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;z#&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;, &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;&amp;#x26;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;str, &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;&amp;#x26;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;count))&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;    {&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;        return&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; NULL&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;    }&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;    int&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; buffer_size &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; (&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;int&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;)count;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#DCBDFB&quot;&gt;    printf&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;Initiailzed Jitter with code size: &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#F47067&quot;&gt;%d&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt; bytes&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#F47067&quot;&gt;\n&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;, buffer_size);&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;    int&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; res &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#DCBDFB&quot;&gt; emit_to_rw_run_from_rx&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(str);&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;    result &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#DCBDFB&quot;&gt; PyLong_FromLong&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(res);&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;    return&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; result;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;}&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;static&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; PyMethodDef JitterMethods&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;[]&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; {&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;	{&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;jit&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;, jitter_jit, METH_VARARGS, &lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;Jit a method at runtime&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;},&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;	{&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;NULL&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;, &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;NULL&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;, &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;0&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;, &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;NULL&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;}&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;};&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;static&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt; struct&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; PyModuleDef jittermodule &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; {&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;	PyModuleDef_HEAD_INIT,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;	&quot;jitter&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;	NULL&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;	-&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;1&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;	JitterMethods&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;};&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;PyMODINIT_FUNC &lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#DCBDFB&quot;&gt;PyInit_jitter&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;void&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;) {&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;	PyObject &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;*&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;m;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;	m &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#DCBDFB&quot;&gt; PyModule_Create&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;&amp;#x26;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;jittermodule);&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;	if&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; (m &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;==&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; NULL&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;		return&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; NULL&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;	JitterError &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#DCBDFB&quot;&gt; PyErr_NewException&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;jitter.error&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;, &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;NULL&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;, &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;NULL&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;);&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#DCBDFB&quot;&gt;	Py_INCREF&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(JitterError);&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#DCBDFB&quot;&gt;	PyModule_AddObject&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(m, &lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;error&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;, JitterError);&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;	return&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; m;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;}&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;int&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#DCBDFB&quot;&gt; main&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;int&lt;/span&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt; argc&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;, &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;char&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt; *&lt;/span&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;argv&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;[]&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;) {&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#DCBDFB&quot;&gt;	PyImport_AppendInittab&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;jitter&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;, PyInit_jitter);&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#DCBDFB&quot;&gt;	Py_SetProgramName&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;argv&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;[&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;0&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;]);&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#DCBDFB&quot;&gt;	Py_Initialize&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;();&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#DCBDFB&quot;&gt;	PyImport_ImportModule&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;jitter&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;);&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;}&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The first part of the code is very similar to the blog linked above. However, the main part is the interface to python.
The JitterMethods defines a jit method which will be exposed. The jitter_jit exposes the entry point to which we actually send the byte code to be jitted. This method takes in a series of bytes and generates the code for it.&lt;/p&gt;
&lt;p&gt;To help with the build I defined a file called build.sh and put the following code in it.&lt;/p&gt;
&lt;pre class=&quot;astro-code astro-code-themes github-light github-dark-dimmed&quot; style=&quot;--shiki-light:#24292e;--shiki-dark:#adbac7;--shiki-light-bg:#fff;--shiki-dark-bg:#22272e; overflow-x: auto;--file-name-offset: -0.75rem;&quot; tabindex=&quot;0&quot; data-language=&quot;bash&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#F69D50&quot;&gt;python3&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt; setup.py&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt; build&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Running this should generate an .so file on linux. You should be able to see it in the build folder.
Copy the .so file to the root folder so its easier to run it or you can run it from the Interactive prompt too.&lt;/p&gt;
&lt;p&gt;So here is an example of how to run it without any other api support.&lt;/p&gt;
&lt;pre class=&quot;astro-code astro-code-themes github-light github-dark-dimmed&quot; style=&quot;--shiki-light:#24292e;--shiki-dark:#adbac7;--shiki-light-bg:#fff;--shiki-dark-bg:#22272e; overflow-x: auto;--file-name-offset: -0.75rem;&quot; tabindex=&quot;0&quot; data-language=&quot;python&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;l &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; [&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;hex&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;0x&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;48&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;), &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;hex&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;0x&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;89&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;), &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;hex&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;0x&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;f8&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;), &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;hex&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;0x&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;48&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;), &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;hex&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;0x&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;83&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;), &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;hex&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;0x&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;c0&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;), &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;hex&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;0x&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;04&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;), &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;hex&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;0x&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;c3&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;)]&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;import&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; binascii&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;b &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; bytes&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;int&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(x, &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;16&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;) &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;for&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; x &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;in&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; l)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;import&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; jitter&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;val &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; jitter.jit(b)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;print&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(val)&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The code is the assembly encoding of moving 50 to eax. That’s the return value from the function.&lt;/p&gt;
&lt;pre class=&quot;astro-code astro-code-themes github-light github-dark-dimmed&quot; style=&quot;--shiki-light:#24292e;--shiki-dark:#adbac7;--shiki-light-bg:#fff;--shiki-dark-bg:#22272e; overflow-x: auto;--file-name-offset: -0.75rem;&quot; tabindex=&quot;0&quot; data-language=&quot;asm&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;mov&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; eax&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;, &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;50&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;ret&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Pretty slick huh !!!&lt;/p&gt;
&lt;h2 id=&quot;pyasm2&quot;&gt;Pyasm2&lt;/h2&gt;
&lt;p&gt;Now we change gears and use an api which will generate the bytecode assembly for us. I used a library called PyAsm2 which is a very small library. However, it wasn’t working for me in raw form so I decided to change it a little bit.
&lt;a href=&quot;https://github.com/ssarangi/pyasm2&quot;&gt;PyAsm2 Github&lt;/a&gt;. The code is originally from jbremer but I made some modifications to it for it to be able to give us back raw bytes. jbremer’s code used to return strings which was a little painful to work with.&lt;/p&gt;
&lt;pre class=&quot;astro-code astro-code-themes github-light github-dark-dimmed&quot; style=&quot;--shiki-light:#24292e;--shiki-dark:#adbac7;--shiki-light-bg:#fff;--shiki-dark-bg:#22272e; overflow-x: auto;--file-name-offset: -0.75rem;&quot; tabindex=&quot;0&quot; data-language=&quot;python&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;from&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; x86 &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;import&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt; *&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;import&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; jitter&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;import&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; binascii&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;insts &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; []&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;insts.append(mov(eax, &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;50&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;))&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;insts.append(ret())&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;x86bytes &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; []&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;for&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; inst &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;in&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; insts:&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;    x86bytes &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;+=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; inst.bytes()&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;x86_bytes &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; bytes&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;int&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(x, &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;16&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;) &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;for&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; x &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;in&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; x86bytes)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;val &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; jitter.jit(x86_bytes)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;print&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;Final return value: &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#F47067&quot;&gt;%s&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt; %&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; val)&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;So that’s it. My main goal is to be able to generate code using PyAsm2 and feed it to this simple jitter. However, I am still far from it. Right now, I am tweaking pyasm2 to understand its architecture and it will take some time before I manage to do it but my project &lt;a href=&quot;https://github.com/ssarangi/spiderjit&quot;&gt;SpiderJIT&lt;/a&gt; is where I am trying to do this.&lt;/p&gt;
&lt;p&gt;I will write some more blog posts on how that is going once I make some significant progress.&lt;/p&gt;</content:encoded><category>python3</category><category>python</category><category>jit</category></item><item><title>A Simple Type Verifier for Python Functions using Decorators</title><link>https://ssarangi.github.io/posts/type-verifier/</link><guid isPermaLink="true">https://ssarangi.github.io/posts/type-verifier/</guid><description>A Simple Type Verifier for Python Functions using Decorators. Type verification is useful in debugging or when code is written to handle only a specific set of types.</description><pubDate>Fri, 08 Jan 2016 00:00:00 GMT</pubDate><content:encoded>&lt;h2 id=&quot;introduction&quot;&gt;Introduction:&lt;/h2&gt;
&lt;p&gt;Often times while debugging or if you come from a typed system world it is useful to have some type verification. What I mean by that is you have written your functions in python assuming they are some particular type, however the caller calls your function with some bizarre arguments which you didn’t expect. Instead of informing the caller that the type passed is incorrect, you crash with a bizarre error message which makes no sense.&lt;/p&gt;
&lt;h2 id=&quot;decorators-come-to-the-rescue&quot;&gt;Decorators come to the rescue:&lt;/h2&gt;
&lt;p&gt;This is where python decorators come in handy. A google search comes up with many helpful results but I have listed one which I particularly like.&lt;/p&gt;
&lt;p&gt;&lt;a href=&quot;http://simeonfranklin.com/blog/2012/jul/1/python-decorators-in-12-steps/&quot;&gt;Python Decorators in 12 steps&lt;/a&gt;&lt;/p&gt;
&lt;h2 id=&quot;usage&quot;&gt;Usage:&lt;/h2&gt;
&lt;p&gt;So the first question is how would my code look without such type verification.&lt;/p&gt;
&lt;pre class=&quot;astro-code astro-code-themes github-light github-dark-dimmed&quot; style=&quot;--shiki-light:#24292e;--shiki-dark:#adbac7;--shiki-light-bg:#fff;--shiki-dark-bg:#22272e; overflow-x: auto;--file-name-offset: -0.75rem;&quot; tabindex=&quot;0&quot; data-language=&quot;python&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;def&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#DCBDFB&quot;&gt; raise_exception&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(actual_type, expected_type):&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;    raise&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; Exception&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;Expected &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#F47067&quot;&gt;%s&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt; to be of type &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#F47067&quot;&gt;%s&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt; %&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; (actual_type, expected_type))&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;def&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#DCBDFB&quot;&gt; old_style_foo&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(a, b):&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;    if&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt; not&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; isinstance&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(a, &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;int&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;):&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;        raise_exception(&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;type&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(a), &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;int&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;    if&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt; not&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; isinstance&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(b, &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;str&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;):&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;        raise_exception(&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;type&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(b), &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;str&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;    print&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#F47067&quot;&gt;%s&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;: &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#F47067&quot;&gt;%s&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt; %&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; (b, a))&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;As you can see here, as the number of arguments increase, this becomes quite tedious to do this kind of checking.
Now lets see how it would look with the verifier.&lt;/p&gt;
&lt;pre class=&quot;astro-code astro-code-themes github-light github-dark-dimmed&quot; style=&quot;--shiki-light:#24292e;--shiki-dark:#adbac7;--shiki-light-bg:#fff;--shiki-dark-bg:#22272e; overflow-x: auto;--file-name-offset: -0.75rem;&quot; tabindex=&quot;0&quot; data-language=&quot;python&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#DCBDFB&quot;&gt;@verify&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;a&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;int&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;, &lt;/span&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;b&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;str&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;def&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#DCBDFB&quot;&gt; foo&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(a, b):&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;    print&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#F47067&quot;&gt;%s&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;: &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#F47067&quot;&gt;%s&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt; %&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; (b, a))&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;We could make the type verification more complex by using Union types too for example something like this.&lt;/p&gt;
&lt;pre class=&quot;astro-code astro-code-themes github-light github-dark-dimmed&quot; style=&quot;--shiki-light:#24292e;--shiki-dark:#adbac7;--shiki-light-bg:#fff;--shiki-dark-bg:#22272e; overflow-x: auto;--file-name-offset: -0.75rem;&quot; tabindex=&quot;0&quot; data-language=&quot;python&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#DCBDFB&quot;&gt;@verify&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;a&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;U(&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;int&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;, &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;str&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;), &lt;/span&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;b&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;str&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;def&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#DCBDFB&quot;&gt; foo&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(a, b):&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;    print&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#F47067&quot;&gt;%s&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;: &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#F47067&quot;&gt;%s&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt; %&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; (b, a))&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;So in this case ‘a’ could be either an int or a string. So the type verifier would check for all the types before raising an error.&lt;/p&gt;
&lt;p&gt;So lets see how the error message would look in the case when we pass in a function call such as below.&lt;/p&gt;
&lt;pre class=&quot;astro-code astro-code-themes github-light github-dark-dimmed&quot; style=&quot;--shiki-light:#24292e;--shiki-dark:#adbac7;--shiki-light-bg:#fff;--shiki-dark-bg:#22272e; overflow-x: auto;--file-name-offset: -0.75rem;&quot; tabindex=&quot;0&quot; data-language=&quot;python&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;foo(&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;saa&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;, &lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;test&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;)&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The error message is “Exception: Expected b to be of type: str but received type: &amp;#x3C;class ‘int’&gt;”&lt;/p&gt;
&lt;p&gt;So lets see the code for implementing this verify decorator.&lt;/p&gt;
&lt;pre class=&quot;astro-code astro-code-themes github-light github-dark-dimmed&quot; style=&quot;--shiki-light:#24292e;--shiki-dark:#adbac7;--shiki-light-bg:#fff;--shiki-dark-bg:#22272e; overflow-x: auto;--file-name-offset: -0.75rem;&quot; tabindex=&quot;0&quot; data-language=&quot;python&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;def&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#DCBDFB&quot;&gt; verify&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(func&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;None&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;, &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;**&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;options):&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;    if&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; func &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;is&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt; not&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; None&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;:&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#6A737D;--shiki-dark:#768390&quot;&gt;        # We received the function on this call, so we can define&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#6A737D;--shiki-dark:#768390&quot;&gt;        # and return the inner function&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;        def&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#DCBDFB&quot;&gt; inner&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;*&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;args, &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;**&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;kwargs):&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;            if&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; len&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(options) &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;==&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; 0&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;:&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;                raise&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; Exception&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;Expected verification arguments&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;            func_code &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; func.&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;__code__&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;            arg_names &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; func_code.co_varnames&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;            for&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; k, v &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;in&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; options.items():&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#6A737D;--shiki-dark:#768390&quot;&gt;                # Find the key in the original function&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;                idx &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; arg_names.index(k)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;                if&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; (&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;len&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(args) &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;&gt;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; idx):&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#6A737D;--shiki-dark:#768390&quot;&gt;                    # get the idx&apos;th arg&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;                    arg &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; args[idx]&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;                else&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;:&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#6A737D;--shiki-dark:#768390&quot;&gt;                    # Find in the keyword args&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;                    if&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; k &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;in&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; kwargs:&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;                        arg &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; kwargs.get(k)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;                if&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; isinstance&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(v, U):&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#6A737D;--shiki-dark:#768390&quot;&gt;                    # Unroll the types to check for multiple types&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;                    types_match &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; False&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;                    for&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; dtype &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;in&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; v.types:&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;                        if&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; isinstance&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(arg, dtype):&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;                            types_match &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; True&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;                    if&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; types_match &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;==&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; False&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;:&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;                        raise&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; Exception&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;Expected &quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt; +&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; str&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(k) &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;+&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt; &quot; to be of type: &quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt; +&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; str&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(v) &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;+&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt; &quot; but received type: &quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt; +&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; str&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;type&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(arg)))&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;                elif&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt; not&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; isinstance&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(arg, v):&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;                    raise&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; Exception&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;Expected &quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt; +&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; str&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(k) &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;+&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt; &quot; to be of type: &quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt; +&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; v.&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;__name__&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt; +&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt; &quot; but received type: &quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt; +&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; str&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;type&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(arg)))&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;            output &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; func(&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;*&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;args, &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;**&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;kwargs)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;            return&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; output&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;        return&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; inner&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;    else&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;:&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#6A737D;--shiki-dark:#768390&quot;&gt;        # We didn&apos;t receive the function on this call, so the return value&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#6A737D;--shiki-dark:#768390&quot;&gt;        # of this call will receive it, and we&apos;re getting the options now.&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;        def&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#DCBDFB&quot;&gt; partial_inner&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(func):&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;            return&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; verify(func, &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;**&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;options)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;        return&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; partial_inner&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The structure above is a standard python decorator structure so I won’t explain the nested function structure. Instead I will focus on the type system. Python will deprecate a function called inspect.getargspec which gives the function arguments directly. So I had to directly figure out what argument is to be mapped from the function.&lt;/p&gt;
&lt;pre class=&quot;astro-code astro-code-themes github-light github-dark-dimmed&quot; style=&quot;--shiki-light:#24292e;--shiki-dark:#adbac7;--shiki-light-bg:#fff;--shiki-dark-bg:#22272e; overflow-x: auto;--file-name-offset: -0.75rem;&quot; tabindex=&quot;0&quot; data-language=&quot;python&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;func_code &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; func.&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;__code__&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;arg_names &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; func_code.co_varnames&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;These 2 lines get the arg names as they appear in function. The next step is to iterate over the argument specification received from the verify decorator.&lt;/p&gt;
&lt;pre class=&quot;astro-code astro-code-themes github-light github-dark-dimmed&quot; style=&quot;--shiki-light:#24292e;--shiki-dark:#adbac7;--shiki-light-bg:#fff;--shiki-dark-bg:#22272e; overflow-x: auto;--file-name-offset: -0.75rem;&quot; tabindex=&quot;0&quot; data-language=&quot;python&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#DCBDFB&quot;&gt;@verify&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;a&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;U(&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;int&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;, &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;str&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;), &lt;/span&gt;&lt;span style=&quot;--shiki-light:#E36209;--shiki-dark:#F69D50&quot;&gt;b&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;str&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;)&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;has to be matched to&lt;/p&gt;
&lt;pre class=&quot;astro-code astro-code-themes github-light github-dark-dimmed&quot; style=&quot;--shiki-light:#24292e;--shiki-dark:#adbac7;--shiki-light-bg:#fff;--shiki-dark-bg:#22272e; overflow-x: auto;--file-name-offset: -0.75rem;&quot; tabindex=&quot;0&quot; data-language=&quot;python&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;foo(&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;saa&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;, &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;5&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;)&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;So the options gives us what ‘a’ and ‘b’ types are supposed to be. We iterate over them to figure out which argument in the caller matches them.&lt;/p&gt;
&lt;pre class=&quot;astro-code astro-code-themes github-light github-dark-dimmed&quot; style=&quot;--shiki-light:#24292e;--shiki-dark:#adbac7;--shiki-light-bg:#fff;--shiki-dark-bg:#22272e; overflow-x: auto;--file-name-offset: -0.75rem;&quot; tabindex=&quot;0&quot; data-language=&quot;python&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;for&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; k, v &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;in&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; options.items():&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#6A737D;--shiki-dark:#768390&quot;&gt;    # Find the key in the original function&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;    idx &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; arg_names.index(k)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;    if&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; (&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;len&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(args) &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;&gt;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; idx):&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#6A737D;--shiki-dark:#768390&quot;&gt;        # get the idx&apos;th arg&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;        arg &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; args[idx]&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;    else&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;:&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#6A737D;--shiki-dark:#768390&quot;&gt;        # Find in the keyword args&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;        if&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; k &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;in&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; kwargs:&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;            arg &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; kwargs.get(k)&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;This code basically achieves this. It maps “saa” to ‘a’ and ‘5’ to ‘b’.
After that the code is pretty straightforward. If its a union type then it iterates over the types and checks each type. If it’s not a union type then it simply checks the base type it gets with the arg.&lt;/p&gt;
&lt;p&gt;So the last thing remaining in this is to see the Union class definition.&lt;/p&gt;
&lt;pre class=&quot;astro-code astro-code-themes github-light github-dark-dimmed&quot; style=&quot;--shiki-light:#24292e;--shiki-dark:#adbac7;--shiki-light-bg:#fff;--shiki-dark-bg:#22272e; overflow-x: auto;--file-name-offset: -0.75rem;&quot; tabindex=&quot;0&quot; data-language=&quot;python&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;class&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#F69D50&quot;&gt; U&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;:&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;    def&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; __init__&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(self, &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;*&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;args):&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;        self&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;.types &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; args&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;    def&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; __str__&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(self):&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;        return&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt; &quot;,&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;.join(&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;self&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;.types)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;    __repr__&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; __str__&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;That’s it. The code is not very complex. I haven’t tested thoroughly with keyword args and default arguments but it shouldn’t be too difficult to extend this or fix anything if you happen to use this code.&lt;/p&gt;
&lt;p&gt;&lt;a href=&quot;https://github.com/ssarangi/python_type_verifier&quot;&gt;Github Link&lt;/a&gt;&lt;/p&gt;</content:encoded><category>python3</category><category>python</category><category>decorators</category><category>type verifier</category></item><item><title>PyVyM - Part 2. Design description</title><link>https://ssarangi.github.io/posts/pyvym-part2/</link><guid isPermaLink="true">https://ssarangi.github.io/posts/pyvym-part2/</guid><description>Description of the design of a python based VM. This post details the model, exec_frame stack and other details sorrounding the implementation.</description><pubDate>Wed, 06 Jan 2016 00:00:00 GMT</pubDate><content:encoded>&lt;h2 id=&quot;github-link&quot;&gt;&lt;a href=&quot;https://github.com/ssarangi/PyVyM&quot;&gt;Github Link&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;If you have completed the Part 1 of this series, which gave an introduction to the overall bytecode structure for the virtual machine, then this is a continuation of that same post. In this post I will explain the overall design of the VM before we proceed any further.&lt;/p&gt;
&lt;h3 id=&quot;vm-architecture&quot;&gt;VM Architecture:&lt;/h3&gt;
&lt;p&gt;&lt;img src=&quot;https://ssarangi.github.io/img/blog/pyvym/vm_arch.png&quot; alt=&quot;VM Architecture&quot; title=&quot;Virtual Machine Architecture&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; width=&quot;742&quot; height=&quot;667&quot;&gt;&lt;/p&gt;
&lt;p&gt;The diagram above shows the two main components present in the VM. An execution frame  (exec frame) and the VM itself which maintains a stack of exec frames.&lt;/p&gt;
&lt;h4 id=&quot;execution-frame&quot;&gt;Execution Frame:&lt;/h4&gt;
&lt;p&gt;The execution frame can be thought of as a secluded object which can execute a piece of code. It maintains its own code object (which it receives from the python bytecode), an IP (also known as instruction pointer), the locals in the current scope and globals. An instruction pointer keeps track of which instruction is currently being executed. So the big question is why do we need an exec frame. The main purpose of an exec frame is to manage scope.&lt;/p&gt;
&lt;p&gt;Scopes occur with different kinds of constructs in a language. For example, a module is the outermost scope so all variables, functions etc defined in module scope appear as globals.
Functions, then provide the next level of scoping abilities. Locals can be defined within functions which need to be maintained. The next level of scoping comes from control flow (also known as blocks generated from if-else-endif, while loops, for loops etc).&lt;/p&gt;
&lt;p&gt;So to manage these different kinds of scopes it is important to manage execution frames. I will give an example below.&lt;/p&gt;
&lt;pre class=&quot;astro-code astro-code-themes github-light github-dark-dimmed&quot; style=&quot;--shiki-light:#24292e;--shiki-dark:#adbac7;--shiki-light-bg:#fff;--shiki-dark-bg:#22272e; overflow-x: auto;--file-name-offset: -0.75rem;&quot; tabindex=&quot;0&quot; data-language=&quot;python&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;def&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#DCBDFB&quot;&gt; foo&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(a, b):&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;  if&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; a &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;&gt;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; b:&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;    a &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; a &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;+&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; b&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;  else&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;:&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;    a &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; a &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;-&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; b&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;  return&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; a&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;def&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#DCBDFB&quot;&gt; main&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;():&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;  foo(&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;3&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;, &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;4&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;main()&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;If you look at the function above it demonstrates a bunch of scopes. For example, the first one is the function foo and main. These 2 functions need to be present in the global scope of the module so that other functions can call it. This is how main is able to call foo by looking into the global scope of the module.&lt;/p&gt;
&lt;p&gt;foo in turn has to maintain locals of a, b and then there is the if and else scope. The if and else scope have to be different execution frames since they could define their own variables in that scope. However, each execution frame has a reference to its parent execution frame.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://ssarangi.github.io/img/blog/pyvym/exec_frame.png&quot; alt=&quot;Execution frame Structure&quot; title=&quot;Execution Frame Structure&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; width=&quot;1312&quot; height=&quot;382&quot;&gt;&lt;/p&gt;
&lt;p&gt;Having this parent-child relationship enables an execution frame to go up the chain to figure out which local is being referenced. There are some more complex cases which I haven’t handled in my code yet. For example, the below code won’t work in the vm right now.&lt;/p&gt;
&lt;pre class=&quot;astro-code astro-code-themes github-light github-dark-dimmed&quot; style=&quot;--shiki-light:#24292e;--shiki-dark:#adbac7;--shiki-light-bg:#fff;--shiki-dark-bg:#22272e; overflow-x: auto;--file-name-offset: -0.75rem;&quot; tabindex=&quot;0&quot; data-language=&quot;python&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;def&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#DCBDFB&quot;&gt; foo&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(a, b):&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;  if&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; a &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;&gt;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; b:&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;    d &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; a &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;+&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; b&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;  else&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;:&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;    d &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; a &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;-&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; b&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;  return&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; d&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;def&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#DCBDFB&quot;&gt; main&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;():&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;  foo(&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;3&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;, &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;4&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;main()&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;In this, I changed the assignment to a new variable ‘d’ which is to be returned. So in this case we have to have this variable in the foo scope rather than in the if else scope. However, there could be variables which are present only in the if-scope or else-scope which are not used outside and have to be present in their respective scopes.&lt;/p&gt;
&lt;h4 id=&quot;program-execution&quot;&gt;Program Execution&lt;/h4&gt;
&lt;p&gt;The diagram below shows the order in which the program given above executes.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://ssarangi.github.io/img/blog/pyvym/program_exec.png&quot; alt=&quot;Program Execution&quot; title=&quot;Program Execution&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; width=&quot;592&quot; height=&quot;3247&quot;&gt;&lt;/p&gt;
&lt;h4 id=&quot;functions&quot;&gt;Functions&lt;/h4&gt;
&lt;pre class=&quot;astro-code astro-code-themes github-light github-dark-dimmed&quot; style=&quot;--shiki-light:#24292e;--shiki-dark:#adbac7;--shiki-light-bg:#fff;--shiki-dark-bg:#22272e; overflow-x: auto;--file-name-offset: -0.75rem;&quot; tabindex=&quot;0&quot; data-language=&quot;python&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;def&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#DCBDFB&quot;&gt; foo&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(a):&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;    b &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; a &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;*&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; 5&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;    print&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;Value of b: &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#F47067&quot;&gt;%s&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt; %&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; b)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;    return&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; b&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;def&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#DCBDFB&quot;&gt; main&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;():&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;    d &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; 5&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;    e &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; 6&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;    m &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; d &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;+&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; e&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;    return&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; foo(m)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;main()&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;This example shows how function objects are created and how they are called. The bytecode for this is shown below.&lt;/p&gt;
&lt;pre class=&quot;astro-code astro-code-themes github-light github-dark-dimmed&quot; style=&quot;--shiki-light:#24292e;--shiki-dark:#adbac7;--shiki-light-bg:#fff;--shiki-dark-bg:#22272e; overflow-x: auto;--file-name-offset: -0.75rem;&quot; tabindex=&quot;0&quot; data-language=&quot;plaintext&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span&gt;1           0 LOAD_CONST               0 (&amp;#x3C;code object foo at 0x7fe7dc4e3f60, file &quot;tests/call_func.py&quot;, line 1&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;            3 LOAD_CONST               1 (&apos;foo&apos;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;            6 MAKE_FUNCTION            0&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;            9 STORE_NAME               0 (foo)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;6          12 LOAD_CONST               2 (&amp;#x3C;code object main at 0x7fe7dc4935d0, file &quot;tests/call_func.py&quot;, line 6&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;           15 LOAD_CONST               3 (&apos;main&apos;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;           18 MAKE_FUNCTION            0&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;           21 STORE_NAME               1 (main)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;12          24 LOAD_NAME                1 (main)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;           27 CALL_FUNCTION            0 (0 positional, 0 keyword pair)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;           30 POP_TOP&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;           31 LOAD_CONST               4 (None)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;           34 RETURN_VALUE&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The first line shows the code object for the foo function when it is created. On line 3 you will notice a MAKE_FUNCTION bytecode. This instructs the vm to create a new function object with the code loaded on the stack and then the name of the function is to be “foo” from line 2.
The STORE_NAME then takes the function object created and stores it in the global dictionary and associates it with the key “foo”.&lt;/p&gt;
&lt;pre class=&quot;astro-code astro-code-themes github-light github-dark-dimmed&quot; style=&quot;--shiki-light:#24292e;--shiki-dark:#adbac7;--shiki-light-bg:#fff;--shiki-dark-bg:#22272e; overflow-x: auto;--file-name-offset: -0.75rem;&quot; tabindex=&quot;0&quot; data-language=&quot;python&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;class&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#F69D50&quot;&gt; Function&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt;Base&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;):&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;    def&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; __init__&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(self, name, defaults, code&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;None&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;):&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;        Base.&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;__init__&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;self&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;        self&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;.__name &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; name&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;        self&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;.__defaults &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; defaults&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;        Base.set_code(&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;self&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;, code)&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The above is the declaration of the function object which is used whenever a MAKE_FUNCTION is encountered.&lt;/p&gt;
&lt;p&gt;Similarly, the same thing happens for the next 4 lines when “main” function is defined.&lt;/p&gt;
&lt;p&gt;The last 5 lines are interesting since they start the execution of the VM. The VM loads the name “main” from the global dictionary and then calls the main function with no arguments. Generally if arguments are present they will be present at this call point. Default arguments are handled when the MAKE_FUNCTION is specified. Going into that will be too much details for this blog, however, running the VM and tracing through it will show how arguments are handled.&lt;/p&gt;
&lt;p&gt;During the Call function, a new execution frame is created and the code for “main” which had been loaded into the main function object is set as the code object for the execution frame.&lt;/p&gt;
&lt;p&gt;Disassembly of the main code object shows this.&lt;/p&gt;
&lt;pre class=&quot;astro-code astro-code-themes github-light github-dark-dimmed&quot; style=&quot;--shiki-light:#24292e;--shiki-dark:#adbac7;--shiki-light-bg:#fff;--shiki-dark-bg:#22272e; overflow-x: auto;--file-name-offset: -0.75rem;&quot; tabindex=&quot;0&quot; data-language=&quot;plaintext&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span&gt;7           0 LOAD_CONST               1 (5)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;            3 STORE_FAST               0 (d)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;8           6 LOAD_CONST               2 (6)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;            9 STORE_FAST               1 (e)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;9          12 LOAD_FAST                0 (d)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;           15 LOAD_FAST                1 (e)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;           18 BINARY_ADD&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;           19 STORE_FAST               2 (m)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;10          22 LOAD_GLOBAL              0 (foo)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;           25 LOAD_FAST                2 (m)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;           28 CALL_FUNCTION            1 (1 positional, 0 keyword pair)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;           31 RETURN_VALUE&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Now the statements here are almost self-explanatory if they are correlated with the python code itself. I will still explain a few lines to get a hang of how to read this bytecode. LOAD_CONST ‘5’ is done which means 5 is added to the top of the stack in the execution frame.&lt;/p&gt;
&lt;p&gt;Each execution frame also maintains a stack which contains the operands which are loaded with the bytecode. As mentioned in the previous blog post, coursera’s course on compilers gives a very good explanation of a stack based machine.&lt;/p&gt;
&lt;p&gt;STORE_FAST, then pops the 5 from the stack and assigns it to the local ‘d’. Similarly, ‘e’ is set to 6. The next step is to load both ‘d’ and ‘e’ and then a BINARY_ADD is done to add ‘d’ + ‘e’ and the result is then added to the stack.
After that, m stores the value of the addition.&lt;/p&gt;
&lt;p&gt;LOAD_GLOBAL ‘foo’ loads the function object associated with ‘foo’ from the module globals and pushes it into the stack. ‘m’ is also pushed into the stack and then another CALL_FUNCTION bytecode instructs the vm to execute foo.
A new execution frame is created for the call to foo and pushed into the vm’s execution frame stack.&lt;/p&gt;
&lt;p&gt;At this point, the code for foo looks like this.&lt;/p&gt;
&lt;pre class=&quot;astro-code astro-code-themes github-light github-dark-dimmed&quot; style=&quot;--shiki-light:#24292e;--shiki-dark:#adbac7;--shiki-light-bg:#fff;--shiki-dark-bg:#22272e; overflow-x: auto;--file-name-offset: -0.75rem;&quot; tabindex=&quot;0&quot; data-language=&quot;plaintext&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span&gt;2           0 LOAD_FAST                0 (a)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;            3 LOAD_CONST               1 (5)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;            6 BINARY_MULTIPLY&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;            7 STORE_FAST               1 (b)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;3          10 LOAD_GLOBAL              0 (print)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;           13 LOAD_CONST               2 (&apos;Value of b: %s&apos;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;           16 LOAD_FAST                1 (b)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;           19 BINARY_MODULO&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;           20 CALL_FUNCTION            1 (1 positional, 0 keyword pair)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;           23 POP_TOP&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;4          24 LOAD_FAST                1 (b)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;           27 RETURN_VALUE&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The ‘exec_frame’ for foo executes this code and then return’s the value ‘b’ into it’s parent’s execution frame as the result of the operation.&lt;/p&gt;
&lt;p&gt;Another thing to notice is the call to the ‘print’ function. The next section explains this.&lt;/p&gt;
&lt;h4 id=&quot;builtins&quot;&gt;Builtins&lt;/h4&gt;
&lt;p&gt;Builtins are functions which we cannot emulate in the vm itself and the underlying machine is supposed to provide functionality for this. Although this definition is not necessarily accurate in terms of compilers, yet it works in this situation. As in the example above, a call to ‘print’ is done. However, we directly use python’s print method in this case.&lt;/p&gt;
&lt;p&gt;The way to implement is simple. When a LOAD_GLOBAL to ‘print’ is encountered, a lookup is done in our globals to find if we have a definition for ‘print’. If this lookup fails, then the vm looks in the builtins module to find. In that case it finds a reference and pushes the function ‘print’ from builtins onto the stack.&lt;/p&gt;
&lt;p&gt;Execution of this ‘builtin’ is done in the CALL_FUNCTION, however, no execution frame is used. Instead a normal python function call is done with the args specified and the result is appended on the result stack.&lt;/p&gt;
&lt;p&gt;Support for lists, dictionaries are added in the same way as this.&lt;/p&gt;
&lt;h4 id=&quot;for-loops&quot;&gt;For Loops&lt;/h4&gt;
&lt;p&gt;As with functions, control flow is also handled in the same way.&lt;/p&gt;
&lt;pre class=&quot;astro-code astro-code-themes github-light github-dark-dimmed&quot; style=&quot;--shiki-light:#24292e;--shiki-dark:#adbac7;--shiki-light-bg:#fff;--shiki-dark-bg:#22272e; overflow-x: auto;--file-name-offset: -0.75rem;&quot; tabindex=&quot;0&quot; data-language=&quot;python&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;def&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#DCBDFB&quot;&gt; for_loop&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(n):&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;    for&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; i &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;in&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; range&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;0&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;, n):&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;        print&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(i)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;def&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#DCBDFB&quot;&gt; main&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;():&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;    for_loop(&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;5&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;main()&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The disassembly for the ‘for_loop’ looks like this.&lt;/p&gt;
&lt;pre class=&quot;astro-code astro-code-themes github-light github-dark-dimmed&quot; style=&quot;--shiki-light:#24292e;--shiki-dark:#adbac7;--shiki-light-bg:#fff;--shiki-dark-bg:#22272e; overflow-x: auto;--file-name-offset: -0.75rem;&quot; tabindex=&quot;0&quot; data-language=&quot;plaintext&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span&gt;2           0 SETUP_LOOP              33 (to 36)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;            3 LOAD_GLOBAL              0 (range)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;            6 LOAD_CONST               1 (0)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;            9 LOAD_FAST                0 (n)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;           12 CALL_FUNCTION            2 (2 positional, 0 keyword pair)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;           15 GET_ITER&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;      &gt;&gt;   16 FOR_ITER                16 (to 35)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;           19 STORE_FAST               1 (i)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;3          22 LOAD_GLOBAL              1 (print)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;           25 LOAD_FAST                1 (i)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;           28 CALL_FUNCTION            1 (1 positional, 0 keyword pair)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;           31 POP_TOP&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;           32 JUMP_ABSOLUTE           16&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;      &gt;&gt;   35 POP_BLOCK&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;      &gt;&gt;   36 LOAD_CONST               0 (None)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;           39 RETURN_VALUE&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The SETUP_LOOP is the main thing to pay attention to. It sets up the for loop in this case. Whenever, the vm encounters the SETUP_LOOP it creates a new Block object and a new execution frame. The Block object stores the code for the loop itself. Also, this execution frame is linked to its parent execution frame so that “locals” lookup could be done.&lt;/p&gt;
&lt;p&gt;LOAD_GLOBAL ‘range’ again loads a builtin with the arguments 0 to n. The result of this range call is pushed onto the stack. Now comes the GET_ITER, which turns the top of the stack to an iterable object and pushes it back onto the stack.&lt;/p&gt;
&lt;p&gt;The FOR_ITER method, takes the iterable object from top of stack and calls the ‘next’ method on this iter. The ‘next’ is also a python function which increments the iterator.&lt;/p&gt;
&lt;p&gt;The rest of the calls are similar until we hit the JUMP_ABSOLUTE instruction. This instruction indicates that our loop has completed execution and we should jump back to the beginning of the loop.&lt;/p&gt;
&lt;h4 id=&quot;while-loop&quot;&gt;While loop&lt;/h4&gt;
&lt;pre class=&quot;astro-code astro-code-themes github-light github-dark-dimmed&quot; style=&quot;--shiki-light:#24292e;--shiki-dark:#adbac7;--shiki-light-bg:#fff;--shiki-dark-bg:#22272e; overflow-x: auto;--file-name-offset: -0.75rem;&quot; tabindex=&quot;0&quot; data-language=&quot;python&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;def&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#DCBDFB&quot;&gt; while_loop&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(n):&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;    counter &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; 0&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;    while&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; counter &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;&amp;#x3C;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; n:&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;        print&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(counter)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;        counter &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;+=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; 1&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;def&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#DCBDFB&quot;&gt; main&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;():&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;    while_loop(&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;5&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;main()&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;For while-loops too, the disassembly is shown below.&lt;/p&gt;
&lt;pre class=&quot;astro-code astro-code-themes github-light github-dark-dimmed&quot; style=&quot;--shiki-light:#24292e;--shiki-dark:#adbac7;--shiki-light-bg:#fff;--shiki-dark-bg:#22272e; overflow-x: auto;--file-name-offset: -0.75rem;&quot; tabindex=&quot;0&quot; data-language=&quot;plaintext&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span&gt;2           0 LOAD_CONST               1 (0)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;            3 STORE_FAST               1 (counter)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;3           6 SETUP_LOOP              36 (to 45)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;      &gt;&gt;    9 LOAD_FAST                1 (counter)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;           12 LOAD_FAST                0 (n)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;           15 COMPARE_OP               0 (&amp;#x3C;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;           18 POP_JUMP_IF_FALSE       44&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;4          21 LOAD_GLOBAL              0 (print)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;           24 LOAD_FAST                1 (counter)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;           27 CALL_FUNCTION            1 (1 positional, 0 keyword pair)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;           30 POP_TOP&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;5          31 LOAD_FAST                1 (counter)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;           34 LOAD_CONST               2 (1)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;           37 INPLACE_ADD&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;           38 STORE_FAST               1 (counter)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;           41 JUMP_ABSOLUTE            9&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;      &gt;&gt;   44 POP_BLOCK&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;      &gt;&gt;   45 LOAD_CONST               0 (None)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;           48 RETURN_VALUE&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;While loops have the same SETUP_LOOP which does the same thing as the for-loop. However, while loops have conditional expression which cause it to break out from the loop.&lt;/p&gt;
&lt;p&gt;The COMPARE_OP and the POP_JUMP_IF_FALSE are the byte codes which enable breaking out of the loop. The logic is exactly the same as For Loops.&lt;/p&gt;
&lt;h4 id=&quot;classes&quot;&gt;Classes:&lt;/h4&gt;
&lt;p&gt;Classes are slightly more complex than the rest of the concepts to be implemented. That is because classes could have member variables, static methods, constructors, special functions etc. Every method of a class gets an implicit argument of the ‘self’ which is an instance of the object itself.&lt;/p&gt;
&lt;p&gt;So lets see how the code looks for this.&lt;/p&gt;
&lt;pre class=&quot;astro-code astro-code-themes github-light github-dark-dimmed&quot; style=&quot;--shiki-light:#24292e;--shiki-dark:#adbac7;--shiki-light-bg:#fff;--shiki-dark-bg:#22272e; overflow-x: auto;--file-name-offset: -0.75rem;&quot; tabindex=&quot;0&quot; data-language=&quot;python&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;class&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#F69D50&quot;&gt; Foo&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;:&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;    def&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; __init__&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(self):&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;        self&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;.member1 &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; 1&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;    def&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; print&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(self):&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;        self&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;.member1 &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;+=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; 5&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;        print&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;Member1: &lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#F47067&quot;&gt;%s&lt;/span&gt;&lt;span style=&quot;--shiki-light:#032F62;--shiki-dark:#96D0FF&quot;&gt;&quot;&lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt; %&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; self&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;.member1)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;def&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#DCBDFB&quot;&gt; main&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;():&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;    foo &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; Foo()&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;    foo.print()&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;main()&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The disassembly for the module looks as below.&lt;/p&gt;
&lt;pre class=&quot;astro-code astro-code-themes github-light github-dark-dimmed&quot; style=&quot;--shiki-light:#24292e;--shiki-dark:#adbac7;--shiki-light-bg:#fff;--shiki-dark-bg:#22272e; overflow-x: auto;--file-name-offset: -0.75rem;&quot; tabindex=&quot;0&quot; data-language=&quot;plaintext&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span&gt;1           0 LOAD_BUILD_CLASS&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;             1 LOAD_CONST               0 (&amp;#x3C;code object Foo at 0x7f8d745c1540, file &quot;tests/simple_class.py&quot;, line 1&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;             4 LOAD_CONST               1 (&apos;Foo&apos;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;             7 MAKE_FUNCTION            0&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;            10 LOAD_CONST               1 (&apos;Foo&apos;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;            13 CALL_FUNCTION            2 (2 positional, 0 keyword pair)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;            16 STORE_NAME               0 (Foo)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt; 9          19 LOAD_CONST               2 (&amp;#x3C;code object main at 0x7f8d745c1660, file &quot;tests/simple_class.py&quot;, line 9&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;            22 LOAD_CONST               3 (&apos;main&apos;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;            25 MAKE_FUNCTION            0&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;            28 STORE_NAME               1 (main)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;13          31 LOAD_NAME                1 (main)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;            34 CALL_FUNCTION            0 (0 positional, 0 keyword pair)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;            37 POP_TOP&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;            38 LOAD_CONST               4 (None)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;            41 RETURN_VALUE&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The LOAD_BUILD_CLASS is the main bytecode which indicates that a class declaration needs to be built. For this reason python 3 already provides a builtin called &lt;strong&gt;build_class&lt;/strong&gt;. However, during the implementation of this vm, I decided to avoid the use of this builtin and write the code myself to understand how to actually build a class declaration.&lt;/p&gt;
&lt;p&gt;So, in our vm this is how it works. When the LOAD_BUILD_CLASS is seen, the STATE of the VM is changed to BUILD_CLASS.&lt;/p&gt;
&lt;p&gt;The LOAD_BUILD_CLASS actually is supposed to load a function which builds the class. That is why it provides the code object for “foo” and does a MAKE_FUNCTION after that. So in our MAKE_FUNCION method, if the state is set to BUILD_CLASS then I create a template object which keeps the declaration of this class.&lt;/p&gt;
&lt;p&gt;The template object looks like this. It contains the name of the class, the code and a pointer to the original module for globals.&lt;/p&gt;
&lt;pre class=&quot;astro-code astro-code-themes github-light github-dark-dimmed&quot; style=&quot;--shiki-light:#24292e;--shiki-dark:#adbac7;--shiki-light-bg:#fff;--shiki-dark-bg:#22272e; overflow-x: auto;--file-name-offset: -0.75rem;&quot; tabindex=&quot;0&quot; data-language=&quot;python&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;class&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#F69D50&quot;&gt; BuildClass&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;:&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;    def&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; __init__&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(self, name, code, config, module):&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;        self&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;.__class_name &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; name&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;        self&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;.__code &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; code.co_code&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;        self&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;.__ip &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; 0&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;        self&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;.__stack &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; []&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;        self&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;.__names &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; code.co_names&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;        self&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;.__constants &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; code.co_consts&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;        self&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;.__module &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; module&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;        self&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;.__klass &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; Class(&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;self&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;.__class_name)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;        self&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;.__klass.code &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; code&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;        self&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;.__vm_state &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; VMState.&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;EXEC&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;        self&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;.__config &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; config&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;class&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#F69D50&quot;&gt; Class&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#6CB6FF&quot;&gt;Base&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;):&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;    def&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; __init__&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(self, name):&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;        Base.&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;__init__&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;self&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;        self&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;.__name &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; name&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;        self&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;.__special_funcs &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; {}&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt;        self&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;.__normal_funcs &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; {}&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The BuildClass is a smaller version of the VM which has support for a few opcodes. So when MAKE_FUNCTION is invoked it creates function objects which are kept in a map in special_funcs and normal_funcs.&lt;/p&gt;
&lt;p&gt;One this class is instantiated, it is stored in the global module with the name of the class as the key. Now lets look at the ‘main’ function which creates an instance of this object.&lt;/p&gt;
&lt;pre class=&quot;astro-code astro-code-themes github-light github-dark-dimmed&quot; style=&quot;--shiki-light:#24292e;--shiki-dark:#adbac7;--shiki-light-bg:#fff;--shiki-dark-bg:#22272e; overflow-x: auto;--file-name-offset: -0.75rem;&quot; tabindex=&quot;0&quot; data-language=&quot;plaintext&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span&gt;10           0 LOAD_GLOBAL              0 (Foo)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;             3 CALL_FUNCTION            0 (0 positional, 0 keyword pair)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;             6 STORE_FAST               0 (foo)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;11           9 LOAD_FAST                0 (foo)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;            12 LOAD_ATTR                1 (print)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;            15 CALL_FUNCTION            0 (0 positional, 0 keyword pair)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;            18 POP_TOP&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;            19 LOAD_CONST               0 (None)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;            22 RETURN_VALUE&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Here, when we do &lt;code&gt;foo = Foo() &lt;/code&gt;, that is where an instance of Foo is created and the constructor is called. So a LOAD_GLOBAL of “Foo” is done and CALL_FUNCTION is done. However, CALL_FUNCTION is calling a Class Object here which means we have to recognize this and implicitly call the &lt;strong&gt;init&lt;/strong&gt; method.&lt;/p&gt;
&lt;p&gt;So LOAD_GLOBAL here creates a new object from the template of the class. The object is a simple object with no methods.&lt;/p&gt;
&lt;pre class=&quot;astro-code astro-code-themes github-light github-dark-dimmed&quot; style=&quot;--shiki-light:#24292e;--shiki-dark:#adbac7;--shiki-light-bg:#fff;--shiki-dark-bg:#22272e; overflow-x: auto;--file-name-offset: -0.75rem;&quot; tabindex=&quot;0&quot; data-language=&quot;python&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;class&lt;/span&gt;&lt;span style=&quot;--shiki-light:#6F42C1;--shiki-dark:#F69D50&quot;&gt; ClassImpl&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;:&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;    def&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; __init__&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;(self):&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;        pass&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The reason for using this ClassImpl is so that we can then populate the class members and functions. This is populated from the BuildClass klass object which stores all the special functions and normal functions.&lt;/p&gt;
&lt;p&gt;Look at the disassembly above. The ‘print’ method from the foo object is loaded. This is loaded from the class object. The ‘print’ function disassembly shows this. LOAD_FAST loads a ‘self’. This is the class object which is pushed onto the stack.&lt;/p&gt;
&lt;p&gt;LOAD_ATTR will load the members of the class object and then executes the functions or retrieves the value of the member if its a variable.&lt;/p&gt;
&lt;pre class=&quot;astro-code astro-code-themes github-light github-dark-dimmed&quot; style=&quot;--shiki-light:#24292e;--shiki-dark:#adbac7;--shiki-light-bg:#fff;--shiki-dark-bg:#22272e; overflow-x: auto;--file-name-offset: -0.75rem;&quot; tabindex=&quot;0&quot; data-language=&quot;plaintext&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span&gt;6           0 LOAD_FAST                0 (self)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;            3 DUP_TOP&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;            4 LOAD_ATTR                0 (member1)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;            7 LOAD_CONST               1 (5)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;           10 INPLACE_ADD&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;           11 ROT_TWO&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;           12 STORE_ATTR               0 (member1)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;7          15 LOAD_GLOBAL              1 (print)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;           18 LOAD_CONST               2 (&apos;Member1: %s&apos;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;           21 LOAD_FAST                0 (self)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;           24 LOAD_ATTR                0 (member1)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;           27 BINARY_MODULO&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;           28 CALL_FUNCTION            1 (1 positional, 0 keyword pair)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;           31 POP_TOP&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;           32 LOAD_CONST               0 (None)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;           35 RETURN_VALUE&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;</content:encoded><category>virtualmachine</category><category>vm</category><category>python3</category><category>python</category></item><item><title>PyVyM - A python based VM for &apos;Python&apos;</title><link>https://ssarangi.github.io/posts/pyvym-introduction/</link><guid isPermaLink="true">https://ssarangi.github.io/posts/pyvym-introduction/</guid><description>A virtual machine for python designed in python. It also has an accompanying debugger which will be described in a later post.</description><pubDate>Tue, 05 Jan 2016 00:00:00 GMT</pubDate><content:encoded>&lt;h2 id=&quot;github-link&quot;&gt;&lt;a href=&quot;https://github.com/ssarangi/PyVyM&quot;&gt;Github Link&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;Finally after thinking and planning for a long time, I decided to start my blog and write about some of my tinkering. So here goes. :)
This blog is about a simple process virtual machine written in python for &lt;b&gt;&lt;i&gt;python&lt;/i&gt;&lt;/b&gt;. There are some very good resources if you want to read up on python based vm’s. Here is a curated list to read about.&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href=&quot;https://pythonguy.wordpress.com/2008/04/17/writing-a-virtual-machine-in-python/&quot;&gt;https://pythonguy.wordpress.com/2008/04/17/writing-a-virtual-machine-in-python/&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href=&quot;http://rsms.me/2012/10/14/sol-a-sunny-little-virtual-machine.html&quot;&gt;http://rsms.me/2012/10/14/sol-a-sunny-little-virtual-machine.html&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href=&quot;https://www.youtube.com/watch?v=HVUTjQzESeo&quot;&gt;https://www.youtube.com/watch?v=HVUTjQzESeo&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;h3 id=&quot;so-what-is-a-process-virtual-machine-&quot;&gt;So what is a ‘Process’ Virtual Machine ?&lt;/h3&gt;
&lt;p&gt;Wikipedia defines a process virtual machine as follows: ‘process virtual machines are designed to execute a single computer program by providing an abstracted and platform-independent program execution environment.’
What does this mean ? It means that we emulate the behavior of the platform to run the program. Generally, VM’s target a specific language and emulate the bahavior of executing each instruction. So think of it as a
glorified interpreter although there are some subtle differences. An interpreter is part of a virtual machine. For example, most of the java virtual machines come with interpreters and many with JIT compilers to accelerate
frequently executed code. Java uses Java Virtual Machine (JVM) thus giving it the flexibility to run across platforms, since the difference in the platform is abstracted by the virtual machine. Similarly, C# uses Common
Language Runtime (CLR) which is also a VM.&lt;/p&gt;
&lt;h4 id=&quot;so-what-have-i-implemented-&quot;&gt;So what have I implemented ?&lt;/h4&gt;
&lt;p&gt;I enjoy writing python so I decided to write a virtual machine which could understand python code and execute it.&lt;/p&gt;
&lt;h4 id=&quot;introduction&quot;&gt;Introduction:&lt;/h4&gt;
&lt;p&gt;&lt;img src=&quot;https://ssarangi.github.io/img/blog/pyvym/overall_structure.png&quot; alt=&quot;Overall Flow&quot; title=&quot;Overall Flow&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; width=&quot;940&quot; height=&quot;121&quot;&gt;&lt;/p&gt;
&lt;p&gt;So the diagram above shows the overall flow of the VM. Python source is compiled by the python builtin compile method. Python generates what is called as bytecode. Essentially bytecode is a assembly version of code generated. It looks and feels like assembly. Python generated bytecode is a stack based machine which means that the operands of every operation are pushed onto a stack and then popped when the instruction is actually to be executed.
A good instruction to stack based machines is presented in the Coursera Compilers course. Take a look if you are interested.&lt;/p&gt;
&lt;h4 id=&quot;python-bytecode-structure&quot;&gt;Python Bytecode structure&lt;/h4&gt;
&lt;p&gt;The youtube link at the top of the page gives an introduction to python bytecode and how they are used. I would rather start by giving examples to show what it means.&lt;/p&gt;
&lt;pre class=&quot;astro-code astro-code-themes github-light github-dark-dimmed&quot; style=&quot;--shiki-light:#24292e;--shiki-dark:#adbac7;--shiki-light-bg:#fff;--shiki-dark-bg:#22272e; overflow-x: auto;--file-name-offset: -0.75rem;&quot; tabindex=&quot;0&quot; data-language=&quot;python&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;a &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; 1&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;b &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#005CC5;--shiki-dark:#6CB6FF&quot;&gt; 2&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt;c &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; a &lt;/span&gt;&lt;span style=&quot;--shiki-light:#D73A49;--shiki-dark:#F47067&quot;&gt;*&lt;/span&gt;&lt;span style=&quot;--shiki-light:#24292E;--shiki-dark:#ADBAC7&quot;&gt; b&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The code below shows the disassembly generated by python.&lt;/p&gt;
&lt;pre class=&quot;astro-code astro-code-themes github-light github-dark-dimmed&quot; style=&quot;--shiki-light:#24292e;--shiki-dark:#adbac7;--shiki-light-bg:#fff;--shiki-dark-bg:#22272e; overflow-x: auto;--file-name-offset: -0.75rem;&quot; tabindex=&quot;0&quot; data-language=&quot;plaintext&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span&gt;1           0 LOAD_CONST               0 (1)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;            3 STORE_NAME               0 (a)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;2           6 LOAD_CONST               1 (2)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;            9 STORE_NAME               1 (b)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;3          12 LOAD_NAME                0 (a)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;           15 LOAD_NAME                1 (b)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;           18 BINARY_MULTIPLY&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;           19 STORE_NAME               2 (c)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;           22 LOAD_CONST               2 (None)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;           25 RETURN_VALUE&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;So lets dissect this code. I have annotated the code below with what they mean. This can be easily achieved
by storing this python code in a file and then read the file. The compile method returns a code object.&lt;/p&gt;
&lt;h4 id=&quot;code-object-fields&quot;&gt;code object fields&lt;/h4&gt;
&lt;ul&gt;
&lt;li&gt;co_argcount:	number of arguments (not including * or ** args)&lt;/li&gt;
&lt;li&gt;co_code:	string of raw compiled bytecode&lt;/li&gt;
&lt;li&gt;co_consts:	tuple of constants used in the bytecode&lt;/li&gt;
&lt;li&gt;co_filename:	name of file in which this code object was created&lt;/li&gt;
&lt;li&gt;co_firstlineno:	number of first line in Python source code&lt;/li&gt;
&lt;li&gt;co_flags:	bitmap: 1=optimized | 2=newlocals | 4=*arg | 8=**arg&lt;/li&gt;
&lt;li&gt;co_lnotab:	encoded mapping of line numbers to bytecode indices&lt;/li&gt;
&lt;li&gt;co_name:	name with which this code object was defined&lt;/li&gt;
&lt;li&gt;co_names:	tuple of names of local variables&lt;/li&gt;
&lt;li&gt;co_nlocals:	number of local variables&lt;/li&gt;
&lt;li&gt;co_stacksize:	virtual machine stack space required&lt;/li&gt;
&lt;li&gt;co_varnames:	tuple of names of arguments and local variables&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;I have annotated the bytecode below.&lt;/p&gt;
&lt;pre class=&quot;astro-code astro-code-themes github-light github-dark-dimmed&quot; style=&quot;--shiki-light:#24292e;--shiki-dark:#adbac7;--shiki-light-bg:#fff;--shiki-dark-bg:#22272e; overflow-x: auto;--file-name-offset: -0.75rem;&quot; tabindex=&quot;0&quot; data-language=&quot;plaintext&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span&gt;1 (LineNo corresponding to source)           0(ByteOffset) LOAD_CONST(Opcode)               0 (From Stack) (1) (Value annotated by dis)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;            3 STORE_NAME               0 (a)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;2           6 LOAD_CONST               1 (2)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;            9 STORE_NAME               1 (b)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;3          12 LOAD_NAME                0 (a)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;           15 LOAD_NAME                1 (b)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;           18 BINARY_MULTIPLY&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;           19 STORE_NAME               2 (c)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;           22 LOAD_CONST               2 (None)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;           25 RETURN_VALUE&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Each instruction loads from a different field depending on the instruction. For example, STORE_NAME 1 means that it picks up the name from co_names[1].&lt;/p&gt;
&lt;p&gt;The python dis module disassembly has a very good introduction on this topic.
&lt;a href=&quot;https://docs.python.org/2/library/dis.html&quot;&gt;https://docs.python.org/2/library/dis.html&lt;/a&gt;&lt;/p&gt;
&lt;h5 id=&quot;trace-the-program&quot;&gt;Trace the program&lt;/h5&gt;
&lt;p&gt;&lt;img src=&quot;https://ssarangi.github.io/img/blog/pyvym/exec_flow.png&quot; alt=&quot;Execution Flow&quot; title=&quot;Execution Flow&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; width=&quot;2074&quot; height=&quot;1157&quot;&gt;&lt;/p&gt;
&lt;pre class=&quot;astro-code astro-code-themes github-light github-dark-dimmed&quot; style=&quot;--shiki-light:#24292e;--shiki-dark:#adbac7;--shiki-light-bg:#fff;--shiki-dark-bg:#22272e; overflow-x: auto;--file-name-offset: -0.75rem;&quot; tabindex=&quot;0&quot; data-language=&quot;plaintext&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span&gt;1           0 LOAD_CONST               0 (1)&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The line loads the value 1 onto the stack.&lt;/p&gt;
&lt;pre class=&quot;astro-code astro-code-themes github-light github-dark-dimmed&quot; style=&quot;--shiki-light:#24292e;--shiki-dark:#adbac7;--shiki-light-bg:#fff;--shiki-dark-bg:#22272e; overflow-x: auto;--file-name-offset: -0.75rem;&quot; tabindex=&quot;0&quot; data-language=&quot;plaintext&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span&gt;            3 STORE_NAME               0 (a)&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Once the 1 is loaded onto the stack, the STORE_NAME first pops the value from the stack and then stores it into the variable ‘a’. Now ‘a’ is stored as a global in the module.
Similarly, ‘b’ is also loaded.&lt;/p&gt;
&lt;pre class=&quot;astro-code astro-code-themes github-light github-dark-dimmed&quot; style=&quot;--shiki-light:#24292e;--shiki-dark:#adbac7;--shiki-light-bg:#fff;--shiki-dark-bg:#22272e; overflow-x: auto;--file-name-offset: -0.75rem;&quot; tabindex=&quot;0&quot; data-language=&quot;plaintext&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span&gt;3          12 LOAD_NAME                0 (a)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;           15 LOAD_NAME                1 (b)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;           18 BINARY_MULTIPLY&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Now, ‘a’, ‘b’ are fetched from the global scope (the module) and then popped from the stack and used in the binary multiply operation. So in this case its ‘2’ * ‘1’ = ‘2’ so ‘2’ is stored back onto the stack.&lt;/p&gt;
&lt;pre class=&quot;astro-code astro-code-themes github-light github-dark-dimmed&quot; style=&quot;--shiki-light:#24292e;--shiki-dark:#adbac7;--shiki-light-bg:#fff;--shiki-dark-bg:#22272e; overflow-x: auto;--file-name-offset: -0.75rem;&quot; tabindex=&quot;0&quot; data-language=&quot;plaintext&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span&gt;           19 STORE_NAME               2 (c)&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;So ‘c’ is now stored in the global scope with the value 2.&lt;/p&gt;
&lt;pre class=&quot;astro-code astro-code-themes github-light github-dark-dimmed&quot; style=&quot;--shiki-light:#24292e;--shiki-dark:#adbac7;--shiki-light-bg:#fff;--shiki-dark-bg:#22272e; overflow-x: auto;--file-name-offset: -0.75rem;&quot; tabindex=&quot;0&quot; data-language=&quot;plaintext&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span&gt;22 LOAD_CONST               2 (None)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;25 RETURN_VALUE&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The last 3 lines indicate that the const ‘None’ is being loaded and that is the value the module is eventually
returning.&lt;/p&gt;
&lt;p&gt;This was a very simple example of how the execution flows. In the next blog we will look at the how the vm executes this same simple program.&lt;/p&gt;</content:encoded><category>virtualmachine</category><category>vm</category><category>python3</category><category>python</category></item></channel></rss>