Neural Network from Scratch in Cangjie: Part 5 - 仓颉从头开始的神经网络：第五部分

研究编程

726人浏览 · 2024-12-16 23:06:45

研究编程 · 2024-12-16 23:06:45 发布

Today, we will implement a loss and accuracy function to be able to observe how our network is performing in its currently untrained state. This is the last step before we move on to optimization and training. Training involves doing multiple forward and backward passes, adjusting weights and biases, and monitoring loss and accuracy. If loss goes down and accuracy goes up after each pass, the network is actually learning from the data.

For multi-class classification tasks like what we are trying to achieve, Categorical Crossentropy (CC) is the most popular loss function. In the case of binary classification with 2 classes, that would be Binary Crossentropy.

These functions analyze the results from the Softmax activation function and compare them to ground truths - the `y` in the dataset. When confidence levels in the Softmax output equals to 1 (when the network is 100% sure about the predictions), loss equals to 0, and vice versa.

Imagine Softmax outputs that look like the following:

let softmaxOutputs = [[0.7, 0.1, 0.2], [0.1, 0.5, 0.4], [0.02, 0.9, 0.08]]

...and the `y` values for that batch of outputs are the following (representing cat, dog, and dog):

let classTargets = [0, 1, 1]

In this case, we have 2 classes: 0 - cat, 1 - dog. If we encode the classes this way (and we do in the sample spiral data that is used in this tutorial series), these classes actually are indexes in the Softmax output that we can use to retrieve corresponding predictions from. In order to ensure consistency, we need to encode the labels appropriately at the data cleaning stage.

We get [0.7, 0.5, 0.9] - the model is 70% sure that the first sample is a dog, 50% sure that the second sample is a cat and so on.

Categorical cross entropy is basically the negative logarithm of each confidence level, or -log(x). And "loss" is the average (mean) negative logarithm across samples. What would be the loss and accuracy for the said sample of 3?

First, we get the confidences.

var confidencesList = ArrayList<Float64>([])
for ((targIdx, distribution) in classTargets |> zip(softmaxOutputs)) {
    confidencesList.append(distribution[targIdx])
}

>>> [0.700000, 0.500000, 0.900000]

Second, we calculate the negative logarithm from the confidences array. Because -log(0) is `inf` (infinity) and -log(1) results in a negative value (and loss cannot be negative), we need to ensure that our confidences are never 0 or 1, otherwise the network throws an error. What we can do is `clip` our values by adding a very small number to a prediction if it is 0 and subtracting the same number from a prediction if it is 1. The book recommends to use 1e-7, which is 0.0000001.

let negLog = confidencesList |> map {i => -log(clamp(i, 1e-7, 1.0 - 1e-7))} |> collectArray

>>> [0.356675, 0.693147, 0.105361]

Because we need to calculate the mean loss across all samples, we first sum up the negative logarithms and then divide the sum by the number of values.

func sum(x: Float64, y: Float64): Float64 {
    return x + y
}
let negLogSum = negLog |> reduce(sum)
let averageLoss = negLogSum.getOrThrow() / Float64(negLog.size)

>>> 0.385061

The mean loss for our 3 samples is ~0.385.

What about the accuracy?

Accuracy is the mean ratio of true predictions vs false predictions across samples. First, we need the so-called "argmax", or indexes of the largest confidences in Softmax outputs.

let indexes = ArrayList<Int64>([])

for (y in yPred) {
    var index = 0
    var value = y[0]

    for ((e, v) in enumerate(y)) {
        if (v > value) {
            index = e
            value = v
        }
    }

    indexes.append(index)
}

For our sample [[0.7, 0.1, 0.2], [0.1, 0.5, 0.4], [0.02, 0.9, 0.08]], argmax is [0, 1, 1]: [0 -> 0.7, 1 -> 0.5, 1 -> 0.9]

Then, we compare it to the `y`. We create a new ArrayList, to which we append the comparison result between `y` and `argmax`. If there is a match, we append 1.0, otherwise append 0.0. Notice that we use Floats instead of Ints; because the resulting accuracy is a floating point number, we need all our numbers to be Floats as well, even the array length.

let values = ArrayList<Float64>([])

for ((i, j) in argmax |> zip(y)) {
    if (i == j) {
        values.append(1.0)
    } else {
        values.append(0.0)
    }
}

let valuesSum = values |> reduce(this.sum)

let accuracy = valuesSum.getOrThrow() / Float64(values.size)

We get the accuracy score by summing up all the values in the intermediate array called `values` and dividing the sum by the number of values in it.

In this case, we have a perfect score of 1, because all of our simulated predictions are correct.

If we choose another sample [[0.7, 0.2, 0.1], [0.5, 0.1, 0.4], [0.02, 0.9, 0.08]], the argmax becomes [0, 0, 1]. When comparing `argmax` [0, 0, 1] and `y` [0, 1, 1], we can see that only 2/3 of the predictions match, which gives us a score of 0.666667.

Now we put everything together and create a general `Loses` class, from which all other possible losses will inherit.

open class Loses {
    public func calculate(output: Array<Array<Float64>>, y: Array<Int64>): (Float64, Float64) {
        // mean loss
        let sampleLoses = this.forward(output, y)
        let sampleLosesSum = sampleLoses |> reduce(this.sum)
        let dataLoss = sampleLosesSum.getOrThrow() / Float64(sampleLoses.size)

        // argmax
        let result = this.argmax(output)
        // accuracy
        let acc = accuracy(result, y)

        return (dataLoss, acc)
    }

    public open func forward(yPred: Array<Array<Float64>>, yTrue: Array<Int64>): Array<Float64> {
        return []
    }

    private func sum(x: Float64, y: Float64): Float64 {
        return x + y
    }

    private func accuracy(argmax: Array<Int64>, y: Array<Int64>): Float64 {
        let values = ArrayList<Float64>([])

        for ((i, j) in argmax |> zip(y)) {
            if (i == j) {
                values.append(1.0)
            } else {
                values.append(0.0)
            }
        }

        let valuesSum = values |> reduce(this.sum)

        let accuracy = valuesSum.getOrThrow() / Float64(values.size)

        return accuracy
    }

    private func argmax(yPred: Array<Array<Float64>>) {
        let indexes = ArrayList<Int64>([])

        for (y in yPred) {
            var index = 0
            var value = y[0]

            for ((e, v) in enumerate(y)) {
                if (v > value) {
                    index = e
                    value = v
                }
            }

            indexes.append(index)
        }

        return indexes.toArray()
    }
}

class Loss_CategoricalCrossentropy <: Loses {
    public override func forward(yPred: Array<Array<Float64>>, yTrue: Array<Int64>): Array<Float64> {
        var confidencesList = ArrayList<Float64>([])
        for ((targIdx, distribution) in yTrue |> zip(yPred)) {
            confidencesList.append(distribution[targIdx])
        }

        let negativeLogLikelyhoods = confidencesList |> map {i => -log(clamp(i, 1e-7, 1.0 - 1e-7))} |> collectArray

        return negativeLogLikelyhoods
    }
}

The `open` keyword suggests that this is an abstract class and cannot be instantiated on its own - it needs to be inherited. Then we define the calculate method, which will be shared across all loss classes. It computes the loss and accuracy of our network. Next, we define a general forward method that is `open` to be modified by an inheriting class, as calculations may differ from one type of loss to another.

Finally, we define our CC class, which inherits from the generic Loses class. We override the forward method with negative logarithm calculations.

Now, the full code and one more forward pass.

import matrix4cj.*
import std.collection.*
import std.random.*
import csv4cj.*
import std.os.posix.*
import std.fs.*
import std.convert.*
import std.math.*

let random = Random(0) // seed = 0

main() {
    let X: Array<Array<Float64>>
    let y: Array<Int64>

    (X, y) = getData()

    let dense1 = Layer_Dense(2, 4, X.size)
    let activation1 = Activation_ReLU()

    let dense2 = Layer_Dense(4, 4, X.size)
    let activation2 = Activation_ReLU()

    let dense3 = Layer_Dense(4, 3, X.size)
    let activation3 = Activation_Softmax()

    let lossFunction = Loss_CategoricalCrossentropy()

    dense1.forward(X)
    activation1.forward(dense1.output)
    println(activation1.output[..5])

    dense2.forward(activation1.output)
    activation2.forward(dense2.output)
    println(activation2.output[..5])

    dense3.forward(activation2.output)
    activation3.forward(dense3.output)
    println(activation3.output[..5])

    let (loss, acc) = lossFunction.calculate(activation3.output, y)

    println()
    println("loss: ${loss}")
    println("acc : ${acc}")
}

open class Loses {
    public func calculate(output: Array<Array<Float64>>, y: Array<Int64>): (Float64, Float64) {
        // mean loss
        let sampleLoses = this.forward(output, y)
        let sampleLosesSum = sampleLoses |> reduce(this.sum)
        let dataLoss = sampleLosesSum.getOrThrow() / Float64(sampleLoses.size)

        // argmax
        let result = this.argmax(output)
        // accuracy
        let acc = accuracy(result, y)

        return (dataLoss, acc)
    }

    public open func forward(yPred: Array<Array<Float64>>, yTrue: Array<Int64>): Array<Float64> {
        return []
    }

    private func sum(x: Float64, y: Float64): Float64 {
        return x + y
    }

    private func accuracy(argmax: Array<Int64>, y: Array<Int64>): Float64 {
        let values = ArrayList<Float64>([])

        for ((i, j) in argmax |> zip(y)) {
            if (i == j) {
                values.append(1.0)
            } else {
                values.append(0.0)
            }
        }

        let valuesSum = values |> reduce(this.sum)

        let accuracy = valuesSum.getOrThrow() / Float64(values.size)

        return accuracy
    }

    private func argmax(yPred: Array<Array<Float64>>) {
        let indexes = ArrayList<Int64>([])

        for (y in yPred) {
            var index = 0
            var value = y[0]

            for ((e, v) in enumerate(y)) {
                if (v > value) {
                    index = e
                    value = v
                }
            }

            indexes.append(index)
        }

        return indexes.toArray()
    }
}

class Loss_CategoricalCrossentropy <: Loses {
    public override func forward(yPred: Array<Array<Float64>>, yTrue: Array<Int64>): Array<Float64> {
        var confidencesList = ArrayList<Float64>([])
        for ((targIdx, distribution) in yTrue |> zip(yPred)) {
            confidencesList.append(distribution[targIdx])
        }

        let negativeLogLikelyhoods = confidencesList |> map {i => -log(clamp(i, 1e-7, 1.0 - 1e-7))} |> collectArray

        return negativeLogLikelyhoods
    }
}

class Activation_Softmax {
    var output: Array<Array<Float64>>

    public init() {
        this.output = []
    }

    public func forward(inputs: Array<Array<Float64>>) {
        let output = ArrayList<Array<Float64>>([])

        for (input in inputs) {
            let maxValue = max(input)
            let subtractedInput = input |> map {i: Float64 => i - maxValue.getOrThrow()} |> collectArray

            let exponentiatedInput = subtractedInput |> map {i => exp(i)} |> collectArray

            let normBase = exponentiatedInput |> reduce(sum)

            // 标准化
            let probabilities = exponentiatedInput |> map {i => i / normBase.getOrThrow()} |> collectArray

            output.append(probabilities)
        }

        this.output = output.toArray()
    }

    private func sum(x: Float64, y: Float64): Float64 {
        return x + y
    }
}

class Activation_ReLU {
    var output: Array<Array<Float64>>

    public init() {
        this.output = []
    }

    public func forward(inputs: Array<Array<Float64>>) {
        let output = ArrayList<Array<Float64>>([])

        for (array in inputs) {
            output.append(maximum(array))
        }

        this.output = output.toArray()
    }

    private func maximum(input: Array<Float64>): Array<Float64> {
        func clip(i: Float64): Float64 {
            if (i > 0.0) {
                return i
            } else {
                return 0.0
            }
        }

        let output = input |> map {i => clip(i)} |> collectArray

        return output
    }
}

class Layer_Dense {
    var weights: Matrix
    var biases: Matrix
    var output: Array<Array<Float64>>

    public init(nInputs: Int64, nNeurons: Int64, batchSize: Int64) {
        this.weights = Matrix(
            Array<Array<Float64>>(nNeurons, {_ => Array<Float64>(nInputs, {_ => random.nextFloat64() * 0.01})}))

        this.biases = Matrix(Array<Array<Float64>>(batchSize, {_ => Array<Float64>(nNeurons, {_ => 0.0})}))
        this.output = []
    }

    public func forward(inputs: Array<Array<Float64>>) {
        this.output = Matrix(inputs).times(this.weights.transpose()).plus(this.biases).getArray()
    }
}

func getData() {
    let yIdx: Int64 = 2
    let X = ArrayList<Array<Float64>>([])
    let y = ArrayList<Int64>([])

    let path: String = getcwd()
    let fileStream = File("${path}/test.csv", OpenOption.Open(true, false))

    //打开文件流
    if (fileStream.canRead()) {
        //创建字符读取的解析流
        let stream = UTF8ReaderStream(fileStream)
        let reader = CSVReader(stream)

        //创建格式化的解析参数
        let format: CSVParseFormat = CSVParseFormat.DEFAULT

        //创建解析器
        let csvParser = CSVParser(reader, format)
        for (csvRecord in csvParser) {
            let values = csvRecord.getValues()
            X.append(Array<Float64>(values[..yIdx].size, {j => Float64.parse(values[..yIdx][j].toString())}))
            y.append(Int64.parse(values[yIdx].toString()))
        }

        fileStream.close()
    }

    return (X.toArray(), y.toArray())
}

With all random weights and biases set to 0, our network has accuracy of 35%, loss of ~1.09, and equal confidence of ~33% that each prediction is correct.

This is our base point from which on we will be gradually improving its accuracy and loss scores. The next tutorial will introduce optimization, or a way to go back and adjust weights and biases to affect the aforementioned scores to help our Cangjie neural network to actually learn from the data.