{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "# Does AI colourisation produce a historical bias when presented with grayscale figures?\n\nIn the 1950s Kodak's *Shirley card* made a white woman the reference for colour printing, so film and printers were calibrated to lighter skin. This project asks whether a modern machine-learning colouriser carries a similar bias. I ran Richard Zhang's pretrained *Colorful Image Colorization* model (2016) in Python with OpenCV, then adapted it to colourise **video, frame by frame**, which it was never designed for.\n\n![Title card: the research question](assets/cc2/research-question.jpg)\n\n*Zhang's model files (`.prototxt`, `.caffemodel`, `pts_in_hull.npy`) come from his GitHub repo and aren't included here.*",
   "id": "cell-0"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "## 1 · Setup\nLibraries, and the first problems I hit.",
   "id": "cell-1"
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": 1,
   "outputs": [],
   "source": "!pip install numpy\n#numpy is the library to handle numbers like arrays and stuff + installed it as np as well so install numpy as np, so\n#its referred to as np the whole time\n#numpy is needed cause images are just pixels so theyre a grid of numbers so we need numpy to change them\n\n!pip install opencv-python\n#openCV does all image processing so its needed to read, show and process images especially to convert them and colourize\n\n!pip install moviepy\n\n\nimport argparse\n\nimport cv2 \n#openCV for all image and video processing \nimport numpy as np\n#numpy is the library to handle numbers like arrays and stuff + installed it as np as well so install numpy as np, so\n#its referred to as np the whole time\n#numpy is needed cause images are just pixels so theyre a grid of numbers so we need numpy to change them\nimport os\n#OS is the system that lets python open files and folders so its important to open the images and external resources used in this\n\n#ISSUES I RAN INTO\n#1. One of the first issues I ran into was file path errors, In the code below while creating variables when pasting the paths I originally mixed\n#forward and backward slashes, and across attempts I used the os.path.join commend very incorrectly, by basically putting in only one argument \n#when that happens it path does absolutely nothing. It's meant to combine folders/files together. \n#when I mixed up the slashes in python /n is used as well, and I somehow mixed that up with my own format. So instead changed it to r\" and forward\n#slashes to fix confusion with the backshashes and the /n. ",
   "id": "cell-2"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "![ModuleNotFoundError: no module named torch](assets/cc2/didnt-work-pytorch.jpg)\n\n**What didn't work first:** the GitHub instructions assumed a full PyTorch environment, which I didn't have. I changed direction and used Zhang's pretrained Caffe model through OpenCV instead.",
   "id": "cell-3"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "## 2 · Zhang's pretrained model files\nWhere the dataset bias would live: the training weights and the 313 colours the model is allowed to choose from.",
   "id": "cell-4"
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": 1,
   "outputs": [],
   "source": "#ZHANGS PRETRAINED MODEL FILES: \n\nPROTOTXT = os.path.join( r\"model\\colorization_deploy_v2.prototxt\") \n#to load the resources, model is the folder that the files are in, three resources \n#need prototext, points and model because its the neural networks architecture ! \n#prototext demonstrates how many layers the neural network has and how they function\n#!!Zhangs model according to his own paper \"Colorful Image Colorization,\" published by Cornel uses class rebalancing which means it prevented ML models \n#to only focus on higher classes, instead increasing data and samples from minorities.\n#the os.path.join is used when different code in different files are there, and its used to run the same code on a different platform, since all 3\n#resouces contain other code in them\n\nMODEL = os.path.join( r\"model\\colorization_release_v2.caffemodel\")\n#caffemodel: Its the training, so how the model was trained by using 1 million photographs from Imagenet dataset, and is the weights/parameters the \n#model learnt during the training. So everything the ML learned about what colors go where, to which objects etc.\n#!! SUPER IMPORTANT !! --> if there is any bias in the colorization process to any race or any data bias, those biases will be here\n#\"and will thus work well for these types of images, but not necessarily for others. We call this the \"dataset bias\" problem.\" -Richard Zhang Website\n\nPOINTS = os.path.join( r\"model\\pts_in_hull.npy\")\n#313 color reference points, the way that zhangs model works is that it doesnt predict every color possibility. \n#It splits the color into AB channels, in the LAB format into 313 colors. The model essentially picks what color from the 313 does each pixel \n#belong to, and classifies it. !! BIAS !! -->  so basically if Zhang chose fewer darker skin variations within the 313 colors, than lighter ones\n#thats a built in bias already just in the variety and shades of color the ML can use. ",
   "id": "cell-5"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "## 3 · Loading the model",
   "id": "cell-6"
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": 1,
   "outputs": [],
   "source": "#ACTUALLY LOADING THE MODEL\n\n#creating variables\nPROTOTXT = r\"model/colorization_deploy_v2.prototxt\"\nMODEL = r\"model/colorization_release_v2.caffemodel\"\nPOINTS = r\"model/pts_in_hull.npy\" \n#POINTS ! is the 313 colour reference, meaning it tells the model all the colours that can be used to colourize\n\nprint(\"Load model\")\n#when colourizing it just loads the text outside the cell so we know that its processing the request while we wait\n\nnet = cv2.dnn.readNetFromCaffe(PROTOTXT, MODEL) \n#loads prototxt and model, into variable called net, net is the model/ entire ML\n#now all the resources that zhang trained are in the memory ready to be used\npts = np.load(POINTS) \n#loads the 313 points into a variable called pts, but used NP / numpy so its loading it into a number array",
   "id": "cell-7"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "## 4 · Class rebalancing\nThe key part for this investigation: how Zhang stops the model defaulting to washed-out greys.",
   "id": "cell-8"
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": 1,
   "outputs": [],
   "source": "#CLASS REBALANCING --> That zhang references on their website \n\n#used to find the layers in the neural network by the name \nclass8 = net.getLayerId(\"class8_ab\")\n#these two specifically take the name of the two layers in the neural network, so finding the \"class8_ab\" or \"conv8_313_rh\" layer\n#this layer chooses what color from the 313 colors does each pixel belong to\nconv8 = net.getLayerId(\"conv8_313_rh\") \n#this layer rebalances, with the rh stands for rebalancing., \n#it creates color rebalancing weights, and by getting the names of the layers, it lets the model modify the layers.\n\npts = pts.transpose().reshape(2, 313, 1, 1) \n#!!!! this is super important to project focus on all the code and explanation on this !!!!\n#!! SUPER KEY !! the 313 colors change to the exact format needed for the model to work. \n#--> so the array originally are 313, 2, meaning theres 313 rows with 2 values each so it follows the AB coordinates\n#Its like an axis but on a color array / grid\n#when we reshape it, 2, 313, 1, 1 it adds two more layers. This is needed because the model that zhang built uses a format that needs 4 number imputs\n#instead of 2. \n#ISSUE: when putting together this, I was using a youtube tutorial, but I forgot this section, and it caused an error, because theres a\n#dimension mismatch error, and the model cant function without the correct format\n\nnet.getLayer(class8).blobs = [pts.astype(\"float32\")]\n#puts the 313 colors into the class8 layer. \n#float32, converts numbers into 32 points using what the caffee file wants. \n\nnet.getLayer(conv8).blobs = [np.full([1, 313], 2.606, dtype=\"float32\")]\n#this now puts the rebalancing weights from above, into the conv8 layer. 2.606 is a number that is refrenced in Zhangs paper, its a rebalancing constant\n#that zhang calculated. This array exists because: normal photographs default to washed out colors like grey more often and only training it on those,\n#the ML model defaults to those colors as well. Zhang writes: \" We account for the class imbalance problem by reweighting the loss of each pixel \n#at train time based on the pixel color rarity\" So the 2.606 was part of the reweighting. ",
   "id": "cell-9"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "## 5 · Colourising an image",
   "id": "cell-10"
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": 1,
   "outputs": [],
   "source": "#IMAGE COLORIZATION !! (finally)\n\n# Load the input image\nimage = cv2.imread(r\"wii blnw small.jpg\")\n#imread means image read, it loads my image as pixels and is stored by its RGB value with each pixel\n#here we can imput the photograph path link so it can load the photgraph we want. \nscaled = image.astype(\"float32\") / 255.0 \n#NORMALISATION PROCESS !! meaning dividing every pixel by 255 because the model works smoother when doing this\n\nlab = cv2.cvtColor(scaled, cv2.COLOR_BGR2LAB) \n#!!IMPORTANT!!! \n#This specifically is really important because it converts RGB values in each pizel into LAB which\n#is Lightness,and then color on the green red axis, and then color on the blue yellow axis.\n#its super important because Black and white images only have lightness, and its the model's job to predict colour / AB.\n#!!SO connecting to the research question, we can ask through this part of the code what is the difference of the predicted AB channels when\n#predicting in a minority for example, darker races vs lighter races. \n\nresized = cv2.resize(lab, (224, 224))\n#this model was pretrained, it was pretrained on images that were 244 x 244, so all photos will be resized to\n#match this instead for no errors. \n#ISSUE: When I first started testing, and used images without resizing it just caused it to crash with dimension mismatch errors. \n#ISSUE: Another issue was that even after resizing, if I had photographs that were too large they would just zoom in or choose a different \n#part of the image that had no people on them. So some photos at the start are very zoomed in, but then later I just resized them easier to make\n#them work. THOUGH, the same thing with videos became a hassle because its more difficult to resize videos. \n\n\nL = cv2.split(resized)[0] \n#Splitting the LAB, so this only focusing on L which is Light. \nL -= 50\n#L - means subtraction so its subtracting 50 from every image, supposed to make the photo more even and AI more accurate.\n\nprint(\"Colorizing the image\") \n#Printing colourizing the image after its loading, so we know when its actually colourizing and the other commands\n#are running. \nnet.setInput(cv2.dnn.blobFromImage(L)) \n#sometimes neural networks require the input to be a BLOB the 4 number array, and so this converts it from 2D to 4D into a BLOB\n#the net everytime just passes the information into the net ready for the next step. \nab = net.forward()[0, :, :, :].transpose((1, 2, 0)) \n#net.forward is the command that runs the model, this line makes it predict the colours thats why \n#theres blank spaces, the .transpose changes the LAB and colours and feeds it into the correct format\n#In the investigation: THIS is where the bias would appear or not\n#this line also reorgers the numbers, so instead of two values on a grid, its a grid of 224 where every pixel holds two values\n#I assume its because its a format thats needed for the model to work.\n\nab = cv2.resize(ab, (image.shape[1], image.shape[0]))\n#because we had to change the image size for the model to work, now the resize resizes the image\n#back to its original shape, and dimentions. image.shape1 = width, while image.shape0 = height of the photo. \n#Its in simple terms basically back to reconstructing the image\n\nL = cv2.split(lab)[0] \n#need to show the LAB variables again because the image size has changned back to its original dimentions\ncolorized = np.concatenate((L[:, :, np.newaxis], ab), axis=2)\n#np concentrates uses numpy to change the original light with the models predicted colours\n#AB back into one image, and np.newaxis adds the new dimentions back again so the images match eachother and dont look weird or different. \n\ncolorized = cv2.cvtColor(colorized, cv2.COLOR_LAB2BGR) \n#VERY important step because, this converts the LAB image into RGB again so its able \n#to be displayed by jupyter. \ncolorized = np.clip(colorized, 0, 1)  \n#clip is a command thats used to to basically make the values stay between 0-1, because some outputs\n#that go outside these numbers can erros.\ncolorized = (255 * colorized).astype(\"uint8\") \n#UNDO's the normalization from before, so undos the math, so it can become a format that openCV can \n#actually display as well. \n\ncv2.imshow(\"Original\", image) \n#SHOW: it just shows the image literally, so it opens a tab first with the original loaded image onto it \ncv2.imshow(\"Colorized\", colorized) \n#then it opens a second tab with the final product thats colourized ! \ncv2.waitKey(0) \n#0 means to wait forever since its on waitkey to see how long it needs to keep the windows open waiting before closing. ",
   "id": "cell-11"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "![Grayscale Wii Mii test image loaded into the model](assets/cc2/input-wii-mii.jpg)\n\n*Test input: a grayscale photo of Wii Miis.*\n\n![Original, black and white, and colourised photos of a stage performance with skin-tone swatches](assets/cc2/result-stage-swatches.jpg)\n\n*Original → black & white → colourised, with skin-tone swatches sampled from each. The colourised version flattens everyone towards the same warm cast.*",
   "id": "cell-12"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "## 6 · My adaptation: colourising video frame by frame",
   "id": "cell-13"
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": 1,
   "outputs": [],
   "source": "#VIDEOOOO COLORIZATION IE. MY ADAPTATION AND ADDITIONS IN CODE ABOVE TO FIT INTO FRAME BY FRAME VIDEO !\n\n#ISSUES I RAN INTO:\n#The original code had the capture = cv2.VideoCapture(\"-- insert video here\") code on the top of the code as a placeholder. \n#I kept it there confused after a while not remembering why it was there, and it kept causing error and failure because when jupyter notebook tried\n#to open it the file had no link, plus the command was ran twice, and it was just a broken line. And because a lot of my cells were different at the\n#start with things imported and installed separately it would sometimes run it and sometimes would cause errors depending on how all of the cells \n#would run together for some reason. \n#I just removed the placeholder, and the extra line at the top as well, and only writing capture once, since I also had capture = on \n#more code below when using different links and references and got confused when first writing it. \n\ncapture = cv2.VideoCapture(\"colorization video 3 bnw.mp4\")\n#this command opens my video file using the file path loaded on jupyter notebook, with cv2.VideoCapture uses opencv to let me see the video one \n#frame at a time, so its frame by frame colorization. \n#THE IMPORTANCE OF VIDEO: video represents the consistentcy of the model on frames, to see if the persons race or chracateristics are stable\n#If there are inconsistencies, then I think it links to maybe data biases that come from either background or objective biases, that maybe the \n#model uses things such as context clues to make desicions on someones idenity and characteristics in race and culture. \nif not capture.isOpened():\n    print(\"Video cant be shown.\")\n    exit()\n    #just an error if theres some issue, in the wrong file path or problems with the file and it cannot be opened it will print the error. \nprint(\"Colorizing the video\") \nframe_count = 0 \n# To see the frame count to see if the video is loading slow or just frozen, before the loop starts\n#Printing colourizing the image after its loading, so we know when its actually colourizing and the other commands\n#are running. \n\nwhile True:\n    return_value, frame = capture.read()\n    if not return_value:\n        break\n    frame_count += 1         \n#counts up every frame to see if its actually loading\n    #if true, it suceeded to load the frame\n    if frame_count > 50: \n        break\n     # stops after 50 so we can only see 50 frames of the video.\n    #ISSUE!: I had issues with the lag on 100 frames especially so I changed it, though every time I run it, it still crashes everytime and the\n    #Kernal has to restart everytime but I assume because its jupyter notebook and video is just a lot more difficult to process. \n    \n    \n    print(\"colorizing frame\", frame_count)  \n# prints the text to see if its actually running\n    \n    \n    #SAME AS SECTION ABOVE FOR VIDEO !! --> refer to notes above\n   \n    scaled = frame.astype(\"float32\") / 255.0 \n    lab = cv2.cvtColor(scaled, cv2.COLOR_BGR2LAB) \n    resized = cv2.resize(lab, (224, 224))\n    L = cv2.split(resized)[0] \n    L -= 50\n\n\n    net.setInput(cv2.dnn.blobFromImage(L)) \n    ab = net.forward()[0, :, :, :].transpose((1, 2, 0)) \n\n    ab = cv2.resize(ab, (frame.shape[1], frame.shape[0])) \n\n    L = cv2.split(lab)[0] \n    colorized = np.concatenate((L[:, :, np.newaxis], ab), axis=2)\n\n    colorized = cv2.cvtColor(colorized, cv2.COLOR_LAB2BGR) \n    colorized = np.clip(colorized, 0, 1) \n    colorized = (255 * colorized).astype(\"uint8\") \n\n#CHANGES TO THE CODE ABOVE\n    cv2.imshow(\"Original\", frame) \n# just shows the image literally, so it opens tab first with the original loaded video onto it \n    cv2.imshow(\"Colorized\", colorized) \n# opens a second tab with the final product thats colourized !! \n    cv2.waitKey(25)   \n#0 means to wait forever since its on waitkey to see how long it needs to keep the windows open waiting before closing,\n#25 allows for every frame to pass instead of pressing the key a bunch of times to go to the next frame, since its not a still image\ncapture.release() \n#This just releases the video once the loop ends. ",
   "id": "cell-14"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "![Annotated code: change capture to frame](assets/cc2/process-capture-to-frame.jpg)\n\n*To make the image code work on video, each step has to run on `frame` (one image from the video), not on `capture` (the whole video).*\n\n![AttributeError: NoneType object has no attribute astype](assets/cc2/didnt-work-attributeerror.jpg)\n\n**What didn't work:** an `AttributeError`, because part of my colourisation code sat *outside* the loop. Jupyter had already finished reading the video, so there was no frame left to process.",
   "id": "cell-15"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "## Findings\n\n- **Less biased than I expected:** skin tones mostly stayed close to the original instead of defaulting to lighter tones.\n- **The problem is the mid-range:** medium brown and olive skin lost its depth and undertones, replaced by a generic tan or yellow, sometimes with splotchy grey patches.\n- **Why:** class rebalancing pushes the model towards vibrant colour, but vibrant isn't the same as accurate. For mid-range light levels it falls back on an average.\n- **In video** the tones mostly held between frames, but the same flattening appeared.\n\nLike the Shirley card, the model isn't excluding anyone on purpose. It creates a standard, and this time the standard happens to be *tan*. When the subject is a person, accuracy matters more than vibrancy, because the colour is part of their identity.",
   "id": "cell-16"
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}