• part one
• part two
• part three
• part four

In this video we look at cherry-pick, a powerful Git command that lets you apply the changes introduced by a specific commit from one branch onto another, without merging or rebasing the entire branch. Think of it as picking a single commit — like a cherry — and grafting it onto a different branch.

In the animation, we start on master with commits M0 and M1, then create a feature branch and add three commits F0, F1, and F2 to it. We then switch back to master and add one more commit M2. At this point the two branches have diverged. Instead of merging or rebasing the whole feature branch, we cherry-pick only commit F1 onto master. Git takes the diff introduced by F1 and replays it on top of M2, producing a new commit that carries the same changes but has a different hash and a different parent.

Notice that the feature branch is left untouched, and commits F0 and F2 are not brought over — only F1 is. This is what makes cherry-pick so useful: you can selectively integrate individual commits across branches, for example to backport a bug fix to a release branch without pulling in unrelated work.

For your convenience, here are the Git commands which appear in the video:

git commit -m 'M0'
git commit -m 'M1'
git branch feature
git checkout feature
git commit -m 'F0'
git commit -m 'F1'
git commit -m 'F2'
git checkout master
git commit -m 'M2'
git cherry-pick F1

Here is the video(there is no audio):

Feel free to drop me a comment or e-mail with your constructive criticism. It will be much appreciated!

Enjoy!

Building on the Git Animated series, I’ve started a new project called “Algorithms Narrated”. This series presents solutions to various algorithmic problems—think LeetCode—by narrating the thought process and steps involved in solving them.

About The Series

Each video shows the Java code and problem statement together with examples and the time complexity of the solution.

All videos are added to the Algorithms Narrated playlist on my YouTube channel.

Subscribe to stay updated, and if you find them helpful, please like and share the series.

Happy coding!

Let’s explore how to use multiple versions of Java in a Maven project By this, I mean seemingly compiling and running a project with different Java versions.

I will be using Java 17, 21 and 25 for this demo. Make sure you have them installed on your machine. You can use SDKMAN! to manage multiple Java versions easily. Additionally, ensure Maven is installed.

As always, there are multiple ways to achieve this. Here, I will demonstrate one approach using Maven Toolchains.

Configure Maven Toolchains

First, create a toolchains.xml file in your Maven configuration directory (~/.m2/ on Unix-based systems or %USERPROFILE%\.m2\ on Windows). This file will define the different Java versions you want to use.

<?xml version="1.0" encoding="UTF-8"?>
<toolchains>
  <toolchain>
    <type>jdk</type>
    <provides>
      <version>17</version>
    </provides>
    <configuration>
      <jdkHome>/Users/vladflore/.sdkman/candidates/java/17.0.12-oracle</jdkHome>
    </configuration>
  </toolchain>

  <toolchain>
    <type>jdk</type>
    <provides>
      <version>21</version>
    </provides>
    <configuration>
      <jdkHome>/Users/vladflore/.sdkman/candidates/java/21.0.3-oracle</jdkHome>
    </configuration>
  </toolchain>

  <toolchain>
    <type>jdk</type>
    <provides>
      <version>25</version>
    </provides>
    <configuration>
      <jdkHome>/Users/vladflore/.sdkman/candidates/java/25.0.1-open</jdkHome>
    </configuration>
  </toolchain>
</toolchains>

This configuration specifies three different JDK versions and their respective installation paths.

Set Up Maven Project

I will be using a new Maven project for this demonstration.

> mvn archetype:generate -DgroupId=tech.vladflore.app -DartifactId=multijava -DarchetypeArtifactId=maven-archetype-quickstart -DarchetypeGroupId=org.apache.maven.archetypes -DinteractiveMode=false && cd multijava

Add this code in the App.java file located in src/main/java/tech/vladflore/app/.

package tech.vladflore.app;

public class App {

    public static void main(String[] args) {
        System.out.println(Runtime.version());
    }
}

Now, let’s see how to compile this project with different Java versions using Maven Toolchains.

Update the pom.xml file by including the Maven Toolchains plugin.

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-toolchains-plugin</artifactId>
  <version>3.2.0</version>
  <executions>
    <execution>
      <goals>
        <goal>toolchain</goal>
      </goals>
    </execution>
  </executions>
  <configuration>
    <toolchains>
      <jdk>
        <version>${java.compiler.version}</version>
      </jdk>
    </toolchains>
  </configuration>
</plugin>

The java.compiler.version property will be set when running Maven commands.

Do not forget to add it to the properties section of your pom.xml.

<properties>
  <maven.compiler.release>${java.compiler.version}</maven.compiler.release>
</properties>

We use maven.compiler.release because it ensures both bytecode level and API compatibility.

Compiling with Different Java Versions

At this point, we can compile the project using different Java versions. We can check the Java version used during compilation by running javap on the compiled class file and inspecting the version.

> mvn clean compile -Djava.compiler.version=17

> cd target/classes
> javap -verbose tech.vladflore.app.App | grep major

This should output major version: 61, which corresponds to Java 17.

Let’s try with the other version as well.

> mvn clean compile -Djava.compiler.version=21
...
> cd target/classes
> javap -verbose tech.vladflore.app.App | grep major
major version: 65

> mvn clean compile -Djava.compiler.version=25
...
> cd target/classes
> javap -verbose tech.vladflore.app.App | grep major
major version: 69

So far so good! We have successfully compiled the project with different Java versions.

Running with Different Java Versions

If I were to run the project now, i.e. run the only Java class I have, I would get the same output regardless of the compilation version, as it would use the default Java version set in my environment.

> mvn clean compile -Djava.compiler.version=17 exec:java -Dexec.mainClass=tech.vladflore.app.App
> mvn clean compile -Djava.compiler.version=21 exec:java -Dexec.mainClass=tech.vladflore.app.App
> mvn clean compile -Djava.compiler.version=25 exec:java -Dexec.mainClass=tech.vladflore.app.App

They would all output the same Java version, i.e. 25.0.1+8-27, because that is the default Java version on my machine.

> java --version
openjdk 25.0.1 2025-10-21
...
> ls -l "$JAVA_HOME"
/Users/vladflore/.sdkman/candidates/java/current -> 25.0.1-open
> sdk current java
Using java version 25.0.1-open

How do we run the project with different Java versions then?

One way is to use the JAVA_HOME environment variable to point to the desired Java version before running the project.

We can do this either by exporting the JAVA_HOME variable in the shell or by letting sdkman handle it for us.

Another way is to use the exec-maven-plugin plugin.

Add this to your pom.xml:

<plugin>
  <groupId>org.codehaus.mojo</groupId>
  <artifactId>exec-maven-plugin</artifactId>
  <version>3.1.0</version>
  <configuration>
    <executable>${java.runtime.home}/bin/java</executable>
    <arguments>
      <argument>-cp</argument>
      <argument>${project.build.outputDirectory}</argument>
      <argument>${main.class}</argument>
    </arguments>
  </configuration>
</plugin>

The java.runtime.home and main.class properties will be set when running Maven commands.

> mvn clean compile -Djava.compiler.version=17 exec:exec -Djava.runtime.home=/Users/vladflore/.sdkman/candidates/java/17.0.12-oracle -Dmain.class=tech.vladflore.app.App

This compiles and runs the project with Java 17.

Let’s compile with 17 and run with 25.

> mvn clean compile -Djava.compiler.version=17 exec:exec -Djava.runtime.home=/Users/vladflore/.sdkman/candidates/java/25.0.1-open -Dmain.class=tech.vladflore.app.App

Let’s compile with 25 and run with 17.

> mvn clean compile -Djava.compiler.version=25 exec:exec -Djava.runtime.home=/Users/vladflore/.sdkman/candidates/java/17.0.12-oracle -Dmain.class=tech.vladflore.app.App
...
💥💥💥

We get the following error:

Error: LinkageError occurred while loading main class tech.vladflore.app.App
java.lang.UnsupportedClassVersionError: tech/vladflore/app/App has been compiled by a more recent version of the Java Runtime (class file version 69.0), this version of the Java Runtime only recognizes class file versions up to 61.0
[ERROR] Command execution failed.

which is expected since we are trying to run a class compiled with a newer Java version on an older Java runtime.

Conclusion

In this post, we explored how to manage multiple Java versions in a Maven project using Maven Toolchains for compilation and the exec-maven-plugin for running the application with different Java runtimes. This approach allows developers to test and run their applications across various Java versions seamlessly.

When you have a large codebase and a bug has been introduced somewhere in the history, finding the exact commit that introduced the bug can be challenging. This is where git bisect comes in handy. It automates the process of binary searching through your commit history to find the offending commit.

How it works

• start the bisect process by running git bisect start
• find a known bad commit (the one with the bug) and mark it as bad using git bisect bad <bad-commit-hash>
• find a known good commit (the one without the bug) and mark it as good using git bisect good <good-commit-hash>
• Git will now checkout a commit halfway between the good and bad commits. You need to test this commit to see if the bug is present
• if the bug is present, mark the commit as bad using git bisect bad
• if the bug is not present, mark the commit as good using git bisect good
• repeat the testing and marking process until Git narrows down to the exact commit that introduced the bug
• once the bad commit is found, you can end the bisect process by running git bisect reset

This might sound a bit tedious, but it can save you a lot of time compared to manually checking each commit. Additionally, you can automate the testing and marking process using git bisect run <script>, where <script> is a script that tests the code and returns an exit code indicating whether the commit is good or bad.

Example

Make sure you have Maven and Java installed. My setup is as follows:

mvn --version
Apache Maven 3.9.11 ...
...
Java version: 25.0.1, vendor: Oracle Corporation, runtime: /Users/vladflore/.sdkman/candidates/java/25.0.1-open
...

Note: You can use SDKMAN to easily manage multiple versions of Java and Maven on your machine. Check it out at sdkman.io.

Let’s create a simple Java project using Maven:

mvn archetype:generate \
  -DgroupId=tech.vladflore.app \
  -DartifactId=bisect \
  -DarchetypeArtifactId=maven-archetype-quickstart \
  -DarchetypeGroupId=org.apache.maven.archetypes \
  -DinteractiveMode=false

Open it in your favorite IDE and implement a simple binary search algorithm in App.java. Then, write some unit tests in AppTest.java.

In App.java, implement the binary search algorithm:

package tech.vladflore.app;

public class App {

    public static int binarySearch(int[] arr, int target) {
        if (arr == null || arr.length == 0) {
            return -1;
        }

        int left = 0;
        int right = arr.length - 1;

        while (left <= right) {
            int mid = left + (right - left) / 2;

            if (arr[mid] == target) {
                return mid;
            } else if (arr[mid] < target) {
                left = mid + 1;
            } else {
                right = mid - 1;
            }
        }

        return -1;
    }
}

In AppTest.java, write unit tests for the binary search method:

package tech.vladflore.app;

import static org.junit.jupiter.api.Assertions.assertEquals;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.CsvSource;

class AppTest {

    @Test
    void testBinarySearch_ElementFound() {
        int[] arr = {1, 3, 5, 7, 9, 11, 13};
        assertEquals(3, App.binarySearch(arr, 7));
    }

    @Test
    void testBinarySearch_ElementNotFound() {
        int[] arr = {1, 3, 5, 7, 9, 11, 13};
        assertEquals(-1, App.binarySearch(arr, 6));
    }

    @Test
    void testBinarySearch_FirstElement() {
        int[] arr = {1, 3, 5, 7, 9, 11, 13};
        assertEquals(0, App.binarySearch(arr, 1));
    }

    @Test
    void testBinarySearch_LastElement() {
        int[] arr = {1, 3, 5, 7, 9, 11, 13};
        assertEquals(6, App.binarySearch(arr, 13));
    }

    @Test
    void testBinarySearch_SingleElement_Found() {
        int[] arr = {5};
        assertEquals(0, App.binarySearch(arr, 5));
    }

    @Test
    void testBinarySearch_SingleElement_NotFound() {
        int[] arr = {5};
        assertEquals(-1, App.binarySearch(arr, 3));
    }

    @Test
    void testBinarySearch_EmptyArray() {
        int[] arr = {};
        assertEquals(-1, App.binarySearch(arr, 5));
    }

    @Test
    void testBinarySearch_NullArray() {
        assertEquals(-1, App.binarySearch(null, 5));
    }

    @Test
    void testBinarySearch_LargeArray() {
        int[] arr = new int[1000];
        for (int i = 0; i < arr.length; i++) {
            arr[i] = i * 2;
        }
        assertEquals(500, App.binarySearch(arr, 1000));
    }

    @ParameterizedTest
    @CsvSource({
        "1, 0",
        "3, 1",
        "5, 2",
        "7, 3",
        "9, 4",
        "11, 5",
        "13, 6"
    })
    void testBinarySearch_ParameterizedAllElements(int target, int expectedIndex) {
        int[] arr = {1, 3, 5, 7, 9, 11, 13};
        assertEquals(expectedIndex, App.binarySearch(arr, target));
    }
}

Run mvn clean test to ensure all tests pass.

Commit everything.

Right now the git history should show one commit.

git log --oneline
827ee9a9aa5451626aec62849bd11f1f440b457b (HEAD -> main) First commit

For the current implementation to work, the array must be sorted in ascending order. Let’s implement the invariant that the array is sorted before performing the binary search. Create a new commit with this change. Add this in App.java:

for (int i = 0; i < arr.length - 1; i++) {
    if (arr[i] > arr[i + 1]) {
        throw new IllegalArgumentException("Array must be sorted in ascending order");
    }
}

And the test for it in AppTest.java:

@Test
void testBinarySearch_UnsortedArray_ThrowsException() {
    int[] arr = {5, 3, 7, 1, 9};
    Assertions.assertThrows(IllegalArgumentException.class, () -> {
        App.binarySearch(arr, 7);
    });
}

Right now the git history should show two commits.

git log --oneline
9a81359b0c2e67b26ffbb9e51ea3cc2ccaa5a38c (HEAD -> main) Implement invariant
827ee9a9aa5451626aec62849bd11f1f440b457b First commit

Now, let’s introduce a bug in the implementation of the binary search algorithm.

Change the condition in the while loop from left <= right to left < right in App.java, and commit the change.

Now the git history should show three commits.

git log --oneline
e7431eacefd1fafbd46e3b276b687e08e5fa987a (HEAD -> main) Introduce bug
9a81359b0c2e67b26ffbb9e51ea3cc2ccaa5a38c Implement invariant
827ee9a9aa5451626aec62849bd11f1f440b457b First commit

Let’s add some more commits to make the history longer. All of them would be bad commits. Use the following git command to add three more empty commits (⚠️ do not do this in a real project):

for i in {2..4}; do git commit --allow-empty -m "Bad commit $i"; done

Now the git history should show six commits.

git log --oneline
345d5f1ace5b93665583ec5d6e33c6a92c5cec95 (HEAD -> main) Bad commit 4
2dbac1937ed41b26103c022b6579bd855932a8d1 Bad commit 3
46f9a561f097690f86bbb1cf52a94f23294d7f41 Bad commit 2
e7431eacefd1fafbd46e3b276b687e08e5fa987a Introduce bug
9a81359b0c2e67b26ffbb9e51ea3cc2ccaa5a38c Implement invariant
827ee9a9aa5451626aec62849bd11f1f440b457b First commit

We want to find the commit that introduced the bug, i.e. e7431ea. We could do this manually, by following the steps outlined above, or we could use git bisect run to automate the process. For this we need some script that would run the tests and return exit code 0 if all tests pass, or 1 if any test fails. Create a file named test.sh with the following content:

#!/bin/bash
mvn clean test

exit_code=$?

if [ $exit_code -ne 0 ];
then
  exit 1
fi

exit 0

Make the script executable by running chmod +x test.sh.

Now, run the following:

git bisect start
git bisect bad HEAD
git bisect good 9a81359b0c2e67b26ffbb
git bisect run ./test.sh

Git will now automatically checkout commits, run the tests using the provided script, and mark commits as good or bad based on the exit code of the script. After a few iterations, it will identify the commit that introduced the bug.

e7431eacefd1fafbd46e3b276b687e08e5fa987a is the first bad commit
commit e7431eacefd1fafbd46e3b276b687e08e5fa987a
Author: Vlad Flore <flore.vlad@gmail.com>
Date:   Wed Jan 7 17:03:19 2026 +0100

    Introduce bug

...
bisect found first bad commit

So now we know that the commit e7431ea is the one that introduced the bug. To end the bisect session and return to the original HEAD, run:

git bisect reset

We can compare the identified bad commit with the previous good commit to see what changed:

git diff 9a81359 e7431ea | cat

diff --git a/src/main/java/tech/vladflore/app/App.java b/src/main/java/tech/vladflore/app/App.java
index 4084581..588b0aa 100644
--- a/src/main/java/tech/vladflore/app/App.java
+++ b/src/main/java/tech/vladflore/app/App.java
@@ -16,7 +16,7 @@ public class App {
         int left = 0;
         int right = arr.length - 1;

-        while (left <= right) {
+        while (left < right) {
             int mid = left + (right - left) / 2;

             if (arr[mid] == target) {

And here is our bug!

Conclusion

git bisect is a powerful tool that can save you a lot of time when trying to find the commit that introduced a bug. By automating the binary search process, it allows you to quickly narrow down the range of commits to investigate, making it easier to identify and fix issues in your codebase.

JDK 25 reached General Availability on 16 September 2025.

Java 25 continues the trend toward safer concurrency, reduced boilerplate, and better ergonomics—especially for modern, cloud-native and virtual-thread-heavy applications.

Some of the most notable features include:

• Scoped Values
• Module Import Declarations
• Compact Source Files and Instance Main Methods
• Flexible Constructor Bodies
• Compact Object Headers

There are many more features and improvements in this release, but in this post, I will highlight just a few, for a comprehensive list, check out the official JDK 25 release notes.

Scoped Values

JEP 506

Scoped Values provide a way to share immutable context data across a well-defined execution scope, without relying on thread-local storage. They are designed for modern Java, especially in the presence of virtual threads and structured concurrency.

Typical use cases include propagating request-scoped data such as user identity, trace IDs, or transaction metadata—without explicitly passing parameters through every method call.

The big picture:

ThreadLocal → thread-bound mutable state (legacy, problematic with virtual threads)

ScopedValue → scope-bound immutable context (modern, safe, structured)

Let’s look at a simple example of how Scoped Values work and contrast them with ThreadLocal. The example below demonstrates how to implement tracing of operations using both approaches.

About tracing:

Tracing is a technique used to monitor and log the execution of operations within an application. It helps in understanding the flow of requests, diagnosing issues, and analyzing performance by recording key events and data points during the execution of code. It relies on passing context information, like trace IDs and/or span IDs, through various layers of the application.

import java.time.Instant;
import java.util.UUID;

public class TracingDemo {

  record TraceContext(String traceId, String spanId) {}

  static final ScopedValue<TraceContext> TRACE = ScopedValue.newInstance();

  public static void main(String[] args) {
    new TracingDemo().handleRequest();
  }

  void handleRequest() {
    TraceContext root = new TraceContext(newTraceId(), "root");

    ScopedValue.where(TRACE, root).run(
            () -> {
              log("Handling request");
              queryDatabase();
              callExternalService();
              log("Request completed");
            }
    );
  }

  void queryDatabase() {
    withSpan(
        "db",
        () -> {
          sleep(300);
          log("Querying database");
        });
  }

  void callExternalService() {
    withSpan(
        "http",
        () -> {
          sleep(500);
          log("Calling external service");
        });
  }

  void withSpan(String spanName, Runnable task) {
    TraceContext parent = TRACE.get();
    TraceContext child = new TraceContext(parent.traceId(), spanName);
    ScopedValue.where(TRACE, child).run(task);
  }

  static void log(String message) {
    TraceContext ctx = TRACE.get();
    System.out.printf(
        "%s [trace=%s span=%s] %s%n", Instant.now(), ctx.traceId(), ctx.spanId(), message);
  }

  static String newTraceId() {
    return "trace-" + UUID.randomUUID();
  }

  static void sleep(long millis) {
    try {
      Thread.sleep(millis);
    } catch (InterruptedException e) {
      throw new RuntimeException(e);
    }
  }
}

Running the code above with java --enable-preview TracingDemo.java produces output similar to the following:

2026-01-02T14:01:18.638861Z [trace=trace-4c3f324f-bc37-4141-a0dc-8062c31423ea span=root] Handling request
2026-01-02T14:01:18.943390Z [trace=trace-4c3f324f-bc37-4141-a0dc-8062c31423ea span=db] Querying database
2026-01-02T14:01:19.452530Z [trace=trace-4c3f324f-bc37-4141-a0dc-8062c31423ea span=http] Calling external service
2026-01-02T14:01:19.453604Z [trace=trace-4c3f324f-bc37-4141-a0dc-8062c31423ea span=root] Request completed

Note that the traceId remains consistent across the different operations, while the spanId changes to reflect the current operation being performed. This demonstrates how Scoped Values can effectively propagate context information through various layers of an application without explicit parameter passing.

Contrasting this with a ThreadLocal approach would involve more boilerplate code. Here’s how the same tracing functionality would look using ThreadLocal:

import java.time.Instant;
import java.util.UUID;

public class TracingDemo {

  record TraceContext(String traceId, String spanId) {}

  static final ThreadLocal<TraceContext> TRACE = new ThreadLocal<>(); // [!code highlight]

  public static void main(String[] args) {
    new TracingDemo().handleRequest();
  }

  void handleRequest() {
    TraceContext root = new TraceContext(newTraceId(), "root");
    TRACE.set(root); // [!code highlight]
    try {
      log("Handling request");
      queryDatabase();
      callExternalService();
      log("Request completed");
    } finally {
      // ⚠️ must remember to clean up, or else trace leaks!
      TRACE.remove(); // [!code highlight]
    }
  }

  void queryDatabase() {
    withSpan(
        "db",
        () -> {
          sleep(300);
          log("Querying database");
        });
  }

  void callExternalService() {
    withSpan(
        "http",
        () -> {
          sleep(500);
          log("Calling external service");
        });
  }

  void withSpan(String spanName, Runnable task) {
    TraceContext parent = TRACE.get();
    TraceContext child = new TraceContext(parent.traceId(), spanName);
    TRACE.set(child); // [!code highlight]
    try {
      task.run();
    } finally {
      // ⚠️ must remember to restore previous context
      TRACE.set(parent); // [!code highlight]
    }
  }

  static void log(String message) {
    TraceContext ctx = TRACE.get();
    System.out.printf(
        "%s [trace=%s span=%s] %s%n", Instant.now(), ctx.traceId(), ctx.spanId(), message);
  }

  static String newTraceId() {
    return "trace-" + UUID.randomUUID();
  }

  static void sleep(long millis) {
    try {
      Thread.sleep(millis);
    } catch (InterruptedException e) {
      throw new RuntimeException(e);
    }
  }
}

This approach produces the same output as the Scoped Values example, but can be more error-prone:

• forgotten cleanup → trace leaks
• exceptions break restoration
• parallel execution loses context
• virtual threads make it worse
• nested spans are fragile

Takeaway: Scoped Values make contextual data propagation explicit, safe, and composable—without the lifecycle hazards of ThreadLocal.

Module Import Declarations

JEP 511

This feature introduces a new way to import all of the packages exported by a module using a single import statement. Two immediate benefits of this feature are:

• avoid the noise of multiple type-import-on-demand declarations (e.g., import com.foo.bar.*)
• beginner-friendly way to start using core Java APIs and third-party libraries and fundamental Java classes without having to learn where they are located in a package hierarchy.

The syntax for a module import declaration is: import module <module-name>;

Let’s look at an example of how to use module import declarations.

import module java.base; // [!code highlight]

public class ModuleImportDemo {
  public static void main(String[] args) {
    var strings = new String[] {"apple", "berry", "citrus"};
    Function<String, String> firstLetterUppercase = s -> s.toUpperCase().substring(0, 1);
    Map<String, String> map =
        Stream.of(strings).collect(Collectors.toMap(firstLetterUppercase, Function.identity()));
    IO.println(map);
  }
}

Without the module import declaration, you would need to either import each type individually or use type-import-on-demand declarations, i.e.:

import java.util.Map;
import java.util.function.Function;
import java.util.stream.Collectors;
import java.util.stream.Stream;

or

import java.util.*;
import java.util.function.*;
import java.util.stream.*;

For compact source files, the java.base module is implicitly imported, which means that the example above could be written like this:

// Compact source file with implicit java.base module import
void main() {
    var strings = new String[] {"apple", "berry", "citrus"};
    Function<String, String> firstLetterUppercase = s -> s.toUpperCase().substring(0, 1);
    Map<String, String> map =
        Stream.of(strings).collect(Collectors.toMap(firstLetterUppercase, Function.identity()));
    IO.println(map);
  }

In case of ambiguous imports, adding a single-type-import declaration resolves the ambiguousity:

import module java.sql;
// import module java.base; -> not needed as it's implicitly imported in compact source files

// without the single-type-import declaration below, Date would be ambiguous
import java.util.Date;

void main() {
    IO.println(new Date());
}

Compact Source Files and Instance Main Methods

JEP 512

The goal of this feature is to make Java more approachable for small programs, scripts, and learning scenarios—without removing the option to write fully structured code.

Do not fret if the terms “compact source files” and “instance main methods” are new to you and maybe somewhat confusing. In essence, this feature introduces a more concise way to write Java programs by allowing you to omit boilerplate code such as class declarations and static main methods.

Basically, we can get from this:

// in a file named HelloWorld.java
public class HelloWorld {
    public static void main(String[] args) {
        System.out.println("Hello, World!");
    }
}

to this:

// in a file with any name, as long as the name is a valid Java class name
// this is a ⚠️ compact source file

void main() { // <-- this is the ⚠️ instance main method
    System.out.println("Hello, World!");
}

Both versions can be run by either compiling them with javac and then running with java, or by running them directly with the source-code launcher.

A “half-way” version is to write the code like this:

class Jep512 {
  void main() { // <-- this is the ⚠️ instance main method
    System.out.println("Hello, World!");
  }
}

This can be run the same as before.

Last two versions can be slightly modified to accept command-line arguments, as well. From void main() to void main(String[] args). Furthermore, the println statement can be simplified to just IO.println("Hello, World!");.

Let’s look at another example:

// in a file named args_printer.java
void main(String[] args) {
  IO.println(Arrays.toString(args));
}

This just works! But, why? How come String, IO and Arrays are available without any imports? The answer is that compact source files implicitly import the java.base module, which exports the java.lang and java.util packages, wherein those types are located.

We could have written it like this:

void main(java.lang.String[] args) {
  java.lang.IO.println(java.util.Arrays.toString(args));
}

To check which packages are exported by the java.base module, you can run the following command:

java --describe-module java.base | grep exports | grep -E "java\.(lang|util)"

which produces the following output:

exports java.lang
# ... other exported packages ...
exports java.util
# ... other exported packages ...

Flexible Constructor Bodies

JEP 513

This feature allows statements to appear before an explicit constructor invocation, i.e., super(...) or this(...).

import module java.base;

public class JEP513 {
  void main() {
    var emp1 = new Employee("John", "Doe", 42, "Engineering");
    IO.println(emp1);
  }
}

class Person {
  final String firstName;
  final String lastName;
  final int age;

  Person(String firstName, String lastName, int age) {
    this.firstName = firstName;
    this.lastName = lastName;
    this.age = age;
  }
}

class Employee extends Person {
  final String department;

  Employee(String firstName, String lastName, int age, String department) {
    if (!List.of("HR", "Sales", "Engineering").contains(department)) {
      throw new IllegalArgumentException("'%s' department not valid.".formatted(department));
    }
    this.department = department;
    super(firstName, lastName, age); // [!code highlight]
  }

  @Override
  public String toString() {
    return "Employee(firstName=%s, lastName=%s, age=%d, department=%s)"
        .formatted(this.firstName, this.lastName, this.age, this.department);
  }
}

Compact Object Headers

JEP 519

The main idea here is to reduce the memory footprint of Java objects by optimizing the layout of object headers in memory. Various experiments demonstrate that enabling compact object headers improves performance:

• 22% less heap space
• 8% less CPU time
• 15% less garbage collections with G1 and Parallel collectors

The feature can be enabled with the JVM option -XX:+CompactObjectHeaders.

$ java -XX:+UseCompactObjectHeaders MyProgram.java

Conclusion

Java 25 continues the shift toward safer concurrency, lower ceremony, and better performance—without abandoning Java’s core strengths. Features like Scoped Values and compact source files show a clear focus on modern workloads and developer ergonomics.

One of the most important things in programming is making HTTP calls to talk with some API. Java has very good support for this, and in this post, we will see how to call some HTTP endpoints using Java’s HttpClient class. This class is part of the Java 11 standard library, so you don’t need any external dependencies to use it.

A closer look at the HTTP client

Besides the HttpClient class, there are a few other classes that we will use in this post:

• HttpRequest: This class represents an HTTP request
• HttpResponse: This class represents an HTTP response
• HttpRequest.BodyPublishers: This class contains static methods to create request bodies
• HttpResponse.BodyHandlers: This class contains static methods to create response bodies

The new HTTP APIs can be found in the java.net.http package.

It’s important to note that the new HTTP client is intended to replace the legacy HttpURLConnection class, which is still part of Java but considered outdated. The modern HttpClient not only simplifies making HTTP requests, but also adds powerful features like asynchronous calls, native WebSocket support, and HTTP/2 compatibility—making it a more robust and developer-friendly option overall.

The endpoint we will be using for the following examples is the JSONPlaceholder API, which is a free online REST API that you can use for testing and prototyping.

The steps are similar for all the requests we will be making:

1. Create an HttpClient instance; can be reused for multiple requests
2. Create an HttpRequest instance
3. Send the request and get the response
4. Process the response, i.e. either return the raw body or parse it into a Java object

Creating an HTTP client

import java.net.http.HttpClient;

try (var client = HttpClient.newHttpClient()) {
    // Use the client to make requests
} catch (IOException | InterruptedException e) {
    throw new RuntimeException(e);
}

Since HttpClient implements java.lang.AutoCloseable we need to close the resource when we are done with it, the easiest way to do this is by wrapping the instantiation with a try-with-resources block.

Creating an HTTP request

GET request

public static final String GET_SINGLE_ENDPOINT = "https://jsonplaceholder.typicode.com/posts/1";

var request = HttpRequest.newBuilder()
        .uri(new URI(GET_SINGLE_ENDPOINT))
        .GET()
        .build();

HttpResponse<String> response = client.send(request, HttpResponse.BodyHandlers.ofString());

String responseBody = response.body();
int responseStatusCode = response.statusCode();

To fetch all the posts, instead of a single post, we can do the following:

public static final String GET_ALL_ENDPOINT = "https://jsonplaceholder.typicode.com/posts";

var request = HttpRequest.newBuilder().uri(new URI(GET_ALL_ENDPOINT)).GET().build();
HttpResponse<String> response = client.send(request, HttpResponse.BodyHandlers.ofString());
String responseBody = response.body();

If the raw response body is a JSON string, we can use a library like Jackson or Gson to parse it into a Java object.

import com.fasterxml.jackson.databind.ObjectMapper;

var objectMapper = new ObjectMapper();
Post[] posts = objectMapper.readValue(responseBody, Post[].class);

var gson = new Gson();
Post[] posts = gson.fromJson(responseBody, Post[].class);

Make sure to add the Jackson or Gson dependency to your pom.xml file:

<dependency>
    <groupId>com.fasterxml.jackson.core</groupId>
    <artifactId>jackson-databind</artifactId>
    <version>2.17.2</version>
</dependency>
<dependency>
    <groupId>com.google.code.gson</groupId>
    <artifactId>gson</artifactId>
    <version>2.13.1</version>
</dependency>

For the following code examples, making the request and getting the raw body is the same.

POST request

private static final String POST_ENDPOINT = "https://jsonplaceholder.typicode.com/posts";

var body = """
        {
        "title": "foo",
        "body": "bar",
        "userId": 1
        }
        """;
var request = HttpRequest.newBuilder()
        .uri(new URI(POST_ENDPOINT))
        .POST(HttpRequest.BodyPublishers.ofString(body))
        .headers("Content-Type", "application/json")
        .build();

For a POST with no body, simply use HttpRequest.BodyPublishers.noBody().

PUT request

private static final String PUT_ENDPOINT = "https://jsonplaceholder.typicode.com/posts/1";

var body = """
        {
        "id":1,
        "title": "FOO",
        "body": "BAR",
        "userId": 1
        }
        """;
var request = HttpRequest.newBuilder()
        .uri(URI.create(PUT_ENDPOINT))
        .PUT(HttpRequest.BodyPublishers.ofString(body))
        .headers("Content-Type", "application/json")
        .build();

PATCH request

private static final String PATCH_ENDPOINT = "https://jsonplaceholder.typicode.com/posts/1";

var body = """
        {
        "title" : "dummy title"
        }
        """;
var request = HttpRequest.newBuilder()
        .uri(new URI(PATCH_ENDPOINT))
        .method("PATCH", HttpRequest.BodyPublishers.ofString(body))
        .headers("Content-Type", "application/json")
        .build();

DELETE request

private static final String DELETE_ENDPOINT = "https://jsonplaceholder.typicode.com/posts/1";

var request = HttpRequest.newBuilder()
        .uri(new URI(DELETE_ENDPOINT))
        .DELETE()
        .build();

Making asynchronous calls

Making asynchronous calls is very similar to making synchronous calls. The only difference is that we need to use the sendAsync method instead of the send method, and deal with the CompletableFuture that is returned.

private static void makeAsyncRequest(HttpClient client, HttpRequest request) {
    // send the request asynchronously
    var future = client.sendAsync(request, HttpResponse.BodyHandlers.ofString())
            // process the response
            .thenApply(App::handleResponse)
            // consume the response, by printing the first 5 posts
            .thenAccept(posts -> Arrays.stream(posts).limit(5).forEach(System.out::println));
    // wait for the future to complete, before proceeding with the program
    future.join();
}

private static Post[] handleResponse(HttpResponse<String> response) {
    int statusCode = response.statusCode();
    System.out.println("response.statusCode() = " + statusCode);

    if (statusCode == 200) {
        try {
            return new ObjectMapper().readValue(response.body(), Post[].class);
        } catch (JsonProcessingException e) {
            throw new RuntimeException(e);
        }
    } else {
        System.out.println("Unsuccessful response.");
    }

    return new Post[]{};
}

Specifying other options

When building the client, we can specify various options, such as:

• the proxy to use: .proxy(ProxySelector.getDefault())
• the redirect policy: .followRedirects(HttpClient.Redirect.ALWAYS)
• the authenticator: .authenticator(auth)
• the executor: .executor(Executors.newFixedThreadPool(Runtime.getRuntime().availableProcessors()))

When building the request, we can specify various options, such as:

• the timeout: .timeout(Duration.ofSeconds(1))
• the headers: .header("Content-Type", "application/json")
• the protocol version: .version(HttpClient.Version.HTTP_2)
• the body publishers: .POST(HttpRequest.BodyPublishers.ofString(body))

There are a couple of other body publishers that we can use, such as:

• HttpRequest.BodyPublishers.ofFile(): body content taken from the file
• HttpRequest.BodyPublishers.ofInputStream(): body content taken from an input stream
• HttpRequest.BodyPublishers.ofByteArray(): body content taken from the byte array

When sending the request we can specify how the response body should be handled, such as:

• HttpResponse.BodyHandlers.ofString(): returns the response body as a string
• HttpResponse.BodyHandlers.ofByteArray(): returns the response body as a byte array
• HttpResponse.BodyHandlers.ofFile(): returns the response body as a file
• HttpResponse.BodyHandlers.ofInputStream(): returns the response body as an input stream
• HttpResponse.BodyHandlers.ofPublisher(): returns the response body as a publisher
• HttpResponse.BodyHandlers.ofLines(): returns the response body as a stream of lines

Authentication

When creating the client, we can specify an Authenticator to use for authentication. The Authenticator is a class that provides a way to authenticate requests. We can extend it, and override the getPasswordAuthentication method to provide the credentials.

import java.net.Authenticator;
import java.net.PasswordAuthentication;

public class MyAuthenticator extends Authenticator {

    @Override
    protected PasswordAuthentication getPasswordAuthentication() {
        return new PasswordAuthentication(
                "username",
                "password".toCharArray()
        );
    }
}

Alternatively, we can set the Authorization header in the request, and use a basic authentication scheme or a bearer token.

// for the basic authentication scheme
var auth = Base64.getEncoder().encodeToString("username:password".getBytes());

var getRequest = HttpRequest.newBuilder(new URI("<endpoint>"))
        // set a bearer token
        .header("Authorization", "Bearer <token>")
        // or set a basic authentication scheme
        .header("Authorization", "Basic " + auth)
        .GET()
        .build();

Conclusion

In this post, we have seen how to make HTTP calls using Java’s HttpClient class. We have seen how to create a client, create a request, and send the request. We have also seen how to make asynchronous calls and specify various options for the client and the request.

Java 24 is not a LTS version. The last LTS version was Java 21, released in September 2023.

In this post, I will tackle some of the new features and improvements that Java 24 brings to the table. Note that some of them might have been introduced in previous versions, but for the sake of completeness, I will include them here.

In my post about Java 17, I mentioned some of the noteworthy features introduced in that version, in this post I will focus on the new features introduced since then.

Note that I will not cover all the features, but only the ones I find interesting. For a complete list of features feel free to check the JDK Project page.

Java 18

UTF-8 by default

This can be summarized as:

Specify UTF-8 as the default charset of the standard Java APIs. With this change, APIs that depend upon the default charset will behave consistently across all implementations, operating systems, locales, and configurations.

– JEP 400: UTF-8 by Default

The implementation of Charset.defaultCharset() will return the UTF-8 charset by default, unless overridden with the system property file.encoding. If the value is COMPAT, the default charset is derived from the native.encoding system property, which depends on the locale and charset of the underlying operating system.

A quick way to see the default charset of the current JDK is with the following command:

java -XshowSettings:properties -version 2>&1 | grep file.encoding

Simple Web Server

The Simple Web Server is a minimal HTTP server for serving a single directory hierarchy, and it can be used via the dedicated command-line tool jwebserver or programmatically via its API.

jwebserver
Binding to loopback by default. For all interfaces use "-b 0.0.0.0" or "-b ::".
Serving <directory> and subdirectories on 127.0.0.1 port 8000
URL http://127.0.0.1:8000/

The same can be achieved programmatically with the following code:

var server = SimpleFileServer.createFileServer(new InetSocketAddress(8080),
        Path.of("/"),
        OutputLevel.VERBOSE);
server.start();

For more information, check the JEP 408: Simple Web Server.

Code Snippets in JavaDoc

JEP 413: Code Snippets in Java API Documentation introduces a new tag, @snippet, that allows you to include code snippets in your JavaDoc comments.

Here are some examples of how to use it. The JavaDoc comment of the class A includes an inline code snippet, and two external code snippets. The external code snippets are the entire class B and the named region of the class B called say-howdy.

/**
 *
 * {@snippet :
 *
 * System.out.println("This is a polite class.");
 *
 * }
 *
 * {@snippet class = B}
 *
 * {@snippet class = B region = say-howdy}
 *
 */
public class A {
    public static void main(String[] args) {
        System.out.println("Hello, A!");
    }
}

public class B {
    public static void main(String[] args) {
        System.out.println("Hello, B!");
        // @start region = say-howdy
        System.out.println("Howdy, B!");
        // @end
    }
}

In order to generate the JavaDoc with the code snippets, we need to run the following command:

javadoc -d docs --snippet-path . A.java

This will generate the JavaDoc in the docs directory. Note the --snippet-path option, which specifies the path to the code snippets, in this case the current source directory.

This is how it looks like:

Java 19

Not to be dismissive of the work done in Java 19, but I will not cover it here, as it does not introduce any final features and of personal interest. Feel free to check the Java 19 release notes for more information.

Java 20

Idem. Please check the Java 20 release notes for more information.

Java 21

This is a LTS version, released in September 2023. Check the Java 21 release notes.

Sequenced Collections

JEP 431: Sequenced Collections introduces new interfaces to represent collections with a defined encounter order. This means that each such collection has a well-defined first element, second element, and so on, up to the last element.

The new interfaces are:

• SequencedCollection -> a collection with a defined encounter order
• SequencedSet -> a set with a defined encounter order
• SequencedMap -> a map with a defined encounter order

The diagrams below show how the new interfaces fit into the existing collections type hierarchy.

graph TD
    Collection --- Set
    Collection --- SequencedCollection
    Collection --- Queue

    Set --- SequencedSet
    SequencedSet --- SortedSet
    SortedSet --- NavigableSet
    SequencedSet --- LinkedHashSet
    Set --- SortedSet

    SequencedCollection --- List
    SequencedCollection --- Deque
    SequencedCollection --- SequencedSet

    Queue --- Deque

    Map --- SequencedMap
    SequencedMap --- SortedMap
    SequencedMap --- LinkedHashMap
    SortedMap --- NavigableMap

    classDef highlight stroke:orange,stroke-width:2px;
    class SequencedCollection highlight;
    class SequencedSet highlight;
    class SequencedMap highlight;

Record Patterns

Record patterns enable the deconstruction of records into their components. This allows you to extract the values of a record in a more concise way. Check the following example:

record Point(int x, int y) {}
record Circle(Point center, int radius) implements Shape {}
record Line(Point start, Point end) implements Shape {}
sealed interface Shape permits Circle, Line {}

private void printShape(Shape s) {
    // the old way
    if (s instanceof Circle) {
        var circle = (Circle) s;
        var txt = "Center: (%d, %d) Radius: %d".formatted(
                circle.center().x(),
                circle.center().y(),
                circle.radius());
        System.out.println(txt);
    }

    // the new way using record patterns
    if (s instanceof Circle(Point(int x, int y), int radius)) {
        var txt = "Center: (%d, %d) Radius: %d".formatted(x, y, radius);
        System.out.println(txt);
    }

    // same as before, but using `var`
    if (s instanceof Circle(Point(var x, var y), var radius)) {
        var txt = "Center: (%d, %d) Radius: %d".formatted(x, y, radius);
        System.out.println(txt);
    }

    // an exaustive(note that `Shape` is sealed) switch expression that makes use of record patterns
    var txt1 = switch (s) {
        case Circle(Point(var x, var y), var radius) -> "Center: (%d, %d) Radius: %d".formatted(x, y, radius);
        case Line(Point(var startX, var startY), Point(var endX, var endY)) ->
            "Start: (%d, %d) End: (%d, %d)".formatted(startX, startY, endX, endY);
    };
    System.out.println(txt1);
}

var circle = new Circle(new Point(1, 2), 3);
var line = new Line(new Point(7, 7), new Point(77, 77));
printShape(circle);
printShape(line);

Running the above code will produce the following output:

Center: (1, 2) Radius: 3
Center: (1, 2) Radius: 3
Center: (1, 2) Radius: 3
Center: (1, 2) Radius: 3
Start: (7, 7) End: (77, 77)

Pattern Matching for Switch

The example above, besides showing the new record patterns, also shows the new pattern matching for switch expressions. This allows you to use patterns in switch expressions, making them more powerful and expressive.

var txt1 = switch (s) {
    case Circle(Point(var x, var y), var radius) -> "Center: (%d, %d) Radius: %d".formatted(x, y, radius);
    case Line(Point(var startX, var startY), Point(var endX, var endY)) ->
        "Start: (%d, %d) End: (%d, %d)".formatted(startX, startY, endX, endY);
};

Let’s see another example:

import java.util.regex.Pattern;

public class AboutSwitchPatternMatching {
    public static void main(String[] args) {
        var colors = "red, blue, green,yellow, RED, Blue, grEEn, YELLow";
        var pattern = Pattern.compile(",\\s*");
        pattern.splitAsStream(colors).map(AboutSwitchPatternMatching::translateColorToGerman)
                .forEach(System.out::println);
        System.out.println(translateColorToGerman(null));
        System.out.println(translateColorToGerman("purple"));
    }

    private static String translateColorToGerman(String colorInEnglish) {
        return switch (colorInEnglish) {
            // string matching ignoring case
            case String s when s.equalsIgnoreCase("red") -> "rot";
            case String s when s.equalsIgnoreCase("blue") -> "blau";
            case String s when s.equalsIgnoreCase("green") -> "gruen";
            case String s when s.equalsIgnoreCase("yellow") -> "gelb";
            // `null` case must be handled separately
            case null -> "Unsupported color:" + colorInEnglish;
            // to make `switch` exhaustive, we need a `default` case
            default -> "Unsupported color:" + colorInEnglish;
            // Note: the two lines above can be combined into one:
            // case null, default -> "Unsupported color:" + colorInEnglish;
        };
    }
}

While the example above works fine, using enums will improve the code substantially.

enum Color {
    RED, BLUE, GREEN, YELLOW, PURPLE

}
// the main code changes slightly
var colors = "red, blue, green,yellow, RED, Blue, grEEn, YELLow";
var pattern = Pattern.compile(",\\s*");
pattern.splitAsStream(colors)
        // otherwise mapping to `Color` will fail for some values
        .map(String::toUpperCase)
        .map(Color::valueOf)
        .map(AboutSwitchPatternMatching::translateColorToGerman)
        .forEach(System.out::println);
System.out.println(translateColorToGerman(null));
System.out.println(translateColorToGerman(Color.PURPLE));

private static String translateColorToGerman(Color colorInEnglish) {
    return switch (colorInEnglish) {
        // this is now an exhaustive `switch`
        case Color.RED -> "rot";
        case Color.BLUE -> "blau";
        case Color.GREEN -> "gruen";
        case Color.YELLOW -> "gelb";
        case Color.PURPLE -> "lila";
        case null -> "Unsupported color:" + colorInEnglish;
    };
}

Another important point to keep in mind is case label dominance:

Object str = "this is a string";
var res = switch (str) {
    case CharSequence cs -> "A sequence of length: " + cs.length();
    case String s -> "A string: " + s; // COMPILATION ERROR: this case label is dominated by one of the preceding case labels, i.e. a `String` is a `CharSequence`
    default -> "Something else";
};
System.out.println(res);

Switching the order of the case labels fixes the compilation error.

Note that changing the definition of str, from Object to String, leads to other compilation errors, duplicate unconditional pattern, for the second case label, and switch has both an unconditional pattern and a default label, for the default label. We now need to remove the default label, and one of the remaining two case labels. By this time, our switch would have only one case label.

Virtual Threads

JEP 444: Virtual Threads introduces Virtual Threads, maybe the most important feature in Java 21. Make sure to go through the JEP, as I will not cover all the aspects.

Virtual threads are lightweight threads that dramatically reduce the effort of writing, maintaining, and observing high-throughput concurrent applications.

– JEP 444: Virtual Threads

Before seeing some code let’s go through some buzzwords related to this feature:

• virtual thread -> an instance of java.lang.Thread that is not tied to a particular OS thread
• M:N scheduling -> virtual threads employ M:N scheduling, where a large number (M) of virtual threads is scheduled to run on a smaller number (N) of OS threads. Other types of scheduling include 1:1, for platform threads, and M:1 for Java’s green threads, that all shared one OS thread
• structured concurrency -> tries to simplify multithreading programming, by treating multiple tasks running in different threads as a single unit of work
• platform thread -> an instance of java.lang.Thread implemented in the traditional way, as a thin wrapper around an OS thread
• carrier thread -> the platform thread to which the scheduler assigns a virtual thread is called the virtual thread’s carrier. A VT can be scheduled on different carriers over the course of its lifetime
• mounting and unmounting -> to run code in a virtual thread, the JDK’s virtual thread scheduler assigns the virtual thread for execution on a platform thread by mounting the virtual thread on a platform thread. A VT can unmount from its carrier, at which point the platform thread is free and can carry a different VT
• pinning -> a VT is pinned to its carrier when it executes code insider a synchronized block or method, or when it executes a native method or a foreign function. This can hinder scalability, but it does not make an application incorrect

import java.util.concurrent.Executors;
import java.util.concurrent.ThreadFactory;

public class AboutVirtualThreads {
    public static void main(String[] args) throws InterruptedException {

        // the work to be done
        Runnable r = () -> {
            var tName = Thread.currentThread().getName();
            var isVirtual = Thread.currentThread().isVirtual();
            System.out.println("Hello from " + tName + ". Am I virtual? " + isVirtual);
        };

        // using a builder via the `.ofVirtual()`
        var t1 = Thread.ofVirtual().name("my-virtual-thread-1").start(r);
        t1.join();

        // convenient method for creating a VT and scheduling it to execute
        var t2 = Thread.startVirtualThread(r);
        t2.setName("my-virtual-thread-2");
        t2.join();

        // using a thread factory to create a VT for each task submitted
        ThreadFactory tf = Thread.ofVirtual().name("my-virtual-thread-", 3).factory();
        try (var executor = Executors.newThreadPerTaskExecutor(tf)) {
            executor.submit(r);
        }

        // executor that starts a new VT for each task
        try (var executor = Executors.newVirtualThreadPerTaskExecutor()) {
            executor.submit(r);
        }

        // creates a platform thread using the new API
        Thread t4 = Thread.ofPlatform().name("my-platform-thread-1").start(r);
        t4.join();
    }
}

The code from above produces the following output:

Hello from my-virtual-thread-1. Am I virtual? true
Hello from my-virtual-thread-2. Am I virtual? true
Hello from my-virtual-thread-3. Am I virtual? true
Hello from . Am I virtual? true
Hello from my-platform-thread-1. Am I virtual? false

The main API differences between virtual and platform threads are:

• the public Thread constructors cannot create virtual threads
• virtual threads are always daemon threads, they cannot be changed to be non-daemon
• virtual threads have a fixed priority, Thread.NORM_PRIORITY, this priority cannot be changed
• virtual threads are not active members of thread groups

Here are some guidelines to keep in mind when using virtual threads:

• one thread per task: as virtual threads are lightweight, there is no need to reuse of pool them, just create a new one for each task
• blocking I/O inside a synchronized block can pin a virtual thread - this limitation has been removed in Java 24, via JEP 491: Synchronize Virtual Threads without Pinning
• minimize thread-local usage: as VTs can be in the millions, excessive use of thread-local variables can lead to memory issues
• VTs are most useful when blocking I/O or some other blocking operations, BlockingQueue.take(), are involved, i.e. the workload is not CPU-bound

Java 22

Foreign Function & Memory API

JEP 454: Foreign Function & Memory API introduces a new API that allows Java programs to interoperate with code and data outside of the Java runtime. This includes calling native functions and accessing native memory.

There is a lot going on in this JEP, so I will not cover it all. Instead, I will show you a simple example of how to use it. Feel free to check the JEP’s page.

import java.lang.foreign.*;
import java.lang.invoke.MethodHandle;
import java.lang.invoke.MethodHandles;

public class StrlenExample {
    public static void main(String[] args) throws Throwable {
        Linker linker = Linker.nativeLinker();
        SymbolLookup stdlib = linker.defaultLookup();

        MemorySegment strlenFunc = stdlib.find("strlen")
                .orElseThrow(() -> new RuntimeException("strlen not found"));

        FunctionDescriptor strlenDesc = FunctionDescriptor.of(ValueLayout.JAVA_LONG, ValueLayout.ADDRESS);

        MethodHandle strlen = linker.downcallHandle(strlenFunc, strlenDesc);

        try (Arena arena = Arena.ofConfined()) {
            MemorySegment cString = arena.allocateFrom("Hello, FFM!");

            long length = (long) strlen.invoke(cString);
            System.out.println("Length: " + length);
        }
    }
}

The code uses Java 22’s Foreign Function & Memory (FFM) API to call the native C strlen function, which calculates the length of a C-style string. It first obtains a linker and locates the strlen symbol from the standard C library. Then, it defines the function signature using a FunctionDescriptor, creates a method handle to invoke it, and allocates native memory for the input string using arena.allocateFrom("Hello, FFM!"). Finally, it calls the native strlen function and prints the resulting length. The use of Arena ensures safe memory management by automatically freeing the memory when done.

Running the code with the command below will print the length of the string:

java --enable-native-access=ALL-UNNAMED StrlenExample.java
Length: 11

The --enable-native-access=ALL-UNNAMED flag is necessary to allow the unnamed module (i.e., code not in a named module) to access native code.

Unnamed Variables and Patterns

JEP 456: Unnamed Variables & Patterns summarizes this enhancement as follows:

Enhance the Java programming language with unnamed variables and unnamed patterns, which can be used when variable declarations or nested patterns are required but never used. Both are denoted by the underscore character, _.

– JEP 456: Unnamed Variables & Patterns

Let’s see some examples:

import java.util.List;

public class AboutUnnamedVarsAndPatterns {
    public static void main(String[] args) {

        var words = List.of("one", "two", "three");
        var counter = 0;
        // we do not do anything with a word
        for (String _ : words) {
            counter++;
        }
        System.out.println(counter);

        var a = 1;
        var b = 2;
        // normally, the next line would be removed, in this case it illustrates a point
        var _ = 3;
        System.out.println(a + b);

        var notANumber = "boom";
        try {
            int _ = Integer.parseInt(notANumber);
        } catch (NumberFormatException _) {
            System.out.println("Bad number: " + notANumber);
        }

        System.out.println(generateText(new Circle(new Point(0, 0), 1)));
        System.out.println(generateText(new Line(new Point(1, 1), new Point(2, 2))));

    }

    private static String generateText(Shape s) {
        return switch (s) {
            // an example of unnamed pattern variable: `var _`, as we are not interested in
            // the circle's radius
            case Circle(Point(var x, var y), var _) -> "Center: (%d, %d)".formatted(x, y);

            // as we are not interested in the second point we can ignore it completely,
            // this is an example of a unnnamed pattern
            case Line(Point(var startX, var startY), _) ->
                "Start: (%d, %d)".formatted(startX, startY);
        };
    }

    record Point(int x, int y) {
    }

    sealed interface Shape permits Circle, Line {
    }

    record Circle(Point center, int radius) implements Shape {
    }

    record Line(Point start, Point end) implements Shape {
    }
}

To summarize, we have:

• for (String _ : words) { -> unnamed variable
• var _ = 3; -> unnamed variable
• catch (NumberFormatException _) { -> unnamed variable
• case Circle(Point(var x, var y), var _) -> -> unnamed pattern variable
• case Line(Point(var startX, var startY), _) -> -> unnamed pattern

Launch Multi-File Source-Code Programs

JEP 458: Launch Multi-File Source-Code Programs enhances the java application launcher to be able to run a program supplied as multiple files of Java source code.

// File: First.java
public class First {
    void run() {
        System.out.println(First.class.getName());
    }
}

// File: Second.java
public class Second {
    void run() {
        System.out.println(Second.class.getName());
    }
}

// File: JEP458.java
public class JEP458 {
    public static void main(String[] args) {

        var first = new First();
        first.run();
        var second = new Second();
        second.run();
    }
}

Running java JEP458.java will print:

First
Second

Java 23

Markdown Documentation Comments

JEP 467: Markdown Documentation Comments introduces a new way to write documentation comments using Markdown syntax.

Let’s have a look at the following example:

public class JEP467 {
    /// ## Start of documentation
    ///
    /// This _documentation_ was written in Markdown.
    ///
    /// - `This`
    /// - is
    /// - cool.
    ///
    /// Simple tables are supported:
    ///
    /// | Latin | Greek |
    /// |-------|-------|
    /// | a     | alpha |
    /// | b     | beta  |
    ///
    /// This is some Java code:
    ///
    /// ```java
    /// System.out.println("Mind blowing");
    /// ```
    ///
    /// ## End of documentation
    public static void main(String[] args) {
        System.out.println(greet("Vlad"));
    }

    /// This method builds a greeting and gives it back.
    ///
    /// @param name the name to use in the greeting
    /// @return a greeting as a string
    public static String greet(String name) {
        return "Hello, %s!".formatted(name);
    }
}

Note the use of the /// prefix for the documentation comments.

Give it a run with the following command:

javadoc -d docs JEP467.java

Java 24

Class-File API

JEP 484: Class-File API provides a standard API for parsing, generating, and transforming Java class files. Take a look at the JEP for more information, as I will not cover it here.

Stream Gatherers

JEP 485: Stream Gatherers enhances the Stream API to support custom intermediate operations, allowing stream pipelines to transform data in ways that are not easily achievable with the existing operations.

The core concept is a gatherer, which represents a transform of the elements of a stream. Gatherers can be used to transform elements in a one-to-one, one-to-many, many-to-one, or many-to-many fashion. They can also track previously seen elements in order to influence the transformation of subsequent elements.

From a technical point of view, there are three elements involved in this API:

• the java.util.stream.Gatherer interface, which models a gatherer
• the gather method in the java.util.stream.Stream interface, which takes a Gatherer as an argument
• the java.util.stream.Gatherers class, which provides a set of built-in gatherers

The java.util.stream.Gatherers class introduces the following built-in gatherers:

• fold - performs an ordered reduction-like transformation
• mapConcurrent - executes a mapping function with a configured level of max concurrency, using virtual threads, preserving the ordering of the stream
• scan - performs an incremental accumulation starting from an initial value, and each subsequent value being obtained by applying the given function to the current value and the next input element
• windowFixed - gathers elements into a window of a fixed size
• windowSliding - gathers elements into windows of a given size, where each subsequent window includes all elements of the previous window, except for the least recent

Let’s see now some examples:

import java.util.List;
import java.util.stream.Gatherers;
import java.util.stream.Stream;

public class AboutGatherers {
    public static void main(String[] args) {
        List<List<Integer>> result1 =
                Stream.of(1, 2, 3, 4, 5, 6, 7, 8).gather(Gatherers.windowFixed(3)).toList();
        System.out.println("result1 = " + result1);

        List<List<Integer>> result2 =
                Stream.of(1, 2, 3, 4, 5, 6, 7, 8).gather(Gatherers.windowSliding(2)).toList();
        System.out.println("result2 = " + result2);

        List<String> result3 =
                Stream.of(1, 2, 3, 4, 5, 6, 7, 8, 9)
                        .gather(
                                Gatherers.scan(() -> "", (string, number) -> string + number)
                        )
                        .toList();
        System.out.println("result3 = " + result3);

        var result4 = Stream.iterate(1, n -> n + 1).limit(9).gather(
                Gatherers.mapConcurrent(3, n -> n * n)
        ).toList();
        System.out.println("result4 = " + result4);

        String result5 =
                Stream.of(1, 2, 3, 4, 5, 6, 7, 8, 9)
                        .gather(
                                Gatherers.fold(() -> "", (string, number) -> string + number)
                        ).findFirst().orElse("");
        System.out.println("result5 = " + result5);
    }
}

The output of the code above is:

result1 = [[1, 2, 3], [4, 5, 6], [7, 8]]
result2 = [[1, 2], [2, 3], [3, 4], [4, 5], [5, 6], [6, 7], [7, 8]]
result3 = [1, 12, 123, 1234, 12345, 123456, 1234567, 12345678, 123456789]
result4 = [1, 4, 9, 16, 25, 36, 49, 64, 81]
result5 = 123456789

We can also write our own gatherers. The following example shows how to implement a custom gatherer that implements the distinct by length operation. This means that we will only keep the first occurrence of each element, based on its length.

public static Gatherer<String, Set<Integer>, String> distinctByLength() {
    return Gatherer.ofSequential(
            HashSet::new,
            (seen, element, downstream) -> {
                if (downstream.isRejecting()) {
                    return false;
                }
                if (seen.add(element.length())) {
                    downstream.push(element);
                }
                return true;
            }
    );
}

Using the custom gatherer is as simple as:

var distinctByLength = Stream.of("a", "bc", "de", "fgh").gather(distinctByLength()).toList();
System.out.println(distinctByLength);

The output of the code above is:

[a, bc, fgh]

Have a look at the Gatherer interface, especially at the ofSequential and of methods, to see in more detail how to implement your own gatherers. In the example above, we used the ofSequential method, which is a convenience method, that calls the of method, passing in the initializer and the integrator. The combiner and the finisher are the default ones.

Synchronized Virtual Threads without Pinning

JEP 491: Synchronize Virtual Threads without Pinning fixes the issue of pinning virtual threads when using synchronized blocks or methods. This means that we can now use synchronized blocks and methods without pinning the virtual thread to its carrier. Check the JEP’s page for the nitty-gritty details.

Conclusion

The Java language is evolving like never before, and the features I touched upon in this post are just a small part of the whole picture. Make sure to check the release notes for each version for a complete picture. The language itself is becoming more expressive and powerful and tackles areas that were previously the main domain of other languages. This is a good thing! It’s a good time to be a Java developer.

References

• Java SE Support Roadmap
• Java 24

This post covers the rest of the modules of the Machine Learning Zoomcamp, including the midterm and the two capstone projects.

Below are the posts for the previous modules:

1. Machine Learning Zoomcamp Module 1 - points received: 9 (7/7 for questions + 2 bonus for learning in public)
2. Machine Learning Zoomcamp Module 2 - points received: 5 (5/6 for questions + 0 bonus for learning in public)
3. Machine Learning Zoomcamp Module 3 - points received: 7 (6/6 for questions + 1 bonus for learning in public)
4. Machine Learning Zoomcamp Module 4 - points received: 6 (6/6 for questions + 0 bonus for learning in public)
5. Machine Learning Zoomcamp Module 5 - points received: 6 (6/6 for questions + 0 bonus for learning in public)

Module 6 - Decision Trees and Ensemble Learning

Points received: 6

The homework code can be found here.

Midterm Project

Passed ✅ w/ 23 points.

The project can be found here.

Module 8 - Neural Networks and Deep Learning

Points received: 6

The homework can be found here.

Module 9 - Serverless Deep Learning

Points received: 6

The homework can be found here.

Module 10 - Kubernetes and TensorFlow Serving

Points received: 7

The homework can be found here.

Capstone 1 Project

Passed ✅ w/ 21 points.

The project can be found here.

Capstone 2 Project

Passed ✅ w/ 24 points.

The project can be found here.

Conclusion

Total score: 127

Positioned 27 of 760 students who have submited at least one homework.

Certificate.

Overall, the course was very well structured and the content was very well presented. The projects were challenging but doable, and the community was very supportive. I would highly recommend this course to anyone interested in getting into machine learning. The course is very hands-on and provides a lot of practical experience.

This post is a recap of the Machine Learning Zoomcamp Module 5.

Below are the posts for the previous modules:

1. Machine Learning Zoomcamp Module 1 - points received: 9 (7/7 for questions + 2 bonus for learning in public)
2. Machine Learning Zoomcamp Module 2 - points received: 5 (5/6 for questions + 0 bonus for learning in public)
3. Machine Learning Zoomcamp Module 3 - points received: 7 (6/6 for questions + 1 bonus for learning in public)
4. Machine Learning Zoomcamp Module 4 - points received: 6 (6/6 for questions + 0 bonus for learning in public)

The gist of the module

This module is all about deploying the machine learning model we built in the previous modules. Specifically, we’ll save the model to a file and then load it and expose it as a web service that can be consumed by others.

Saving and loading the model

We can use the pickle library to save the model to a file and load it back.

Saving the model and the dictionary vectorizer:

import pickle

with open('model.bin', 'wb') as model_file_out:
    pickle.dump(model, model_file_out)

with open('dv.bin', 'wb') as dv_file_out:
    pickle.dump(dv, dv_file_out)

Loading the model and the dictionary vectorizer:

import pickle

with open('model.bin', 'rb') as model_file_in:
    model = pickle.load(model_file_in)

with open('dv.bin', 'rb') as dv_file_in:
    dv = pickle.load(dv_file_in)

If the pickle library is not installed, we can install it using pip install pickle-mixin.

Building a web service

We can use the flask library to build the web service and expose the model.

import pickle

from sklearn.feature_extraction import DictVectorizer
from sklearn.linear_model import LogisticRegression
from flask import Flask
from flask import request

app = Flask('prediction-service')
model_file = "model.bin"
dv_file = "dv.bin"


@app.route('/predict', methods=['POST'])
def predict():
    customer = request.get_json()

    with open(model_file, 'rb') as model_file_in:
       model = pickle.load(model_file_in)

    with open(dv_file, 'rb') as dv_file_in:
       dv = pickle.load(dv_file_in)

    X = dv.transform([customer])
    prediction = model.predict_proba(X)[0,1]
    result = {'prediction': float(prediction)}

    return result

if __name__ == "__main__":
    app.run(debug=True, host='0.0.0.0', port=9696)

With the code above in a file called prediction_service.py, we can run the web service by running:

python prediction_service.py

While the code above is a simple example, it’s not recommended to use it in production. For production, we should consider using a production-ready web server (WSGI) like gunicorn. This is as simple as running:

gunicorn --bind 0.0.0.0:9696 prediction_service:app

To install gunicorn, we can run pip install gunicorn.

Containerization

Dependencies management

We can use pipenv to manage the dependencies for the web service.

Installing the dependencies is as easy as running:

pipenv install <dependency>

This will create a Pipfile and a Pipfile.lock file that we can use to install the dependencies in a different environment.

Create a Dockerfile

We can use Docker to containerize the web service.

# Use the specified base image
FROM svizor/zoomcamp-model:3.11.5-slim

# Set the working directory inside the container
WORKDIR /app

# Copy Pipfile and Pipfile.lock to the working directory
COPY ["Pipfile", "Pipfile.lock", "./"]

# Install pipenv
RUN pip install pipenv

# Install dependencies defined in Pipfile.lock
RUN pipenv install --deploy --system

# Copy Python scripts and model files to the working directory
COPY ["*.py", "model.bin", "dv.bin", "./"]

# Expose port 9696 for the web service
EXPOSE 9696

# Define the entry point for the container
ENTRYPOINT ["gunicorn", "--bind", "0.0.0.0:9696", "prediction_service:app"]

Conclusion

We have seen how to export and load a machine learning model and dictionary vectorizer, together with how to build a web service exposing an endpoint for making predictions and how to containerize the web service.

The homework code can be found here.

This post is a recap of the Machine Learning Zoomcamp Module 4.

Below are the posts for the previous modules:

1. Machine Learning Zoomcamp Module 1 - points received: 9 (7/7 for questions + 2 bonus for learning in public)
2. Machine Learning Zoomcamp Module 2 - points received: 5 (5/6 for questions + 0 bonus for learning in public)
3. Machine Learning Zoomcamp Module 3 - points received: 7 (6/6 for questions + 1 bonus for learning in public)

The gist of the module

This module focuses on the evaluation of classification models. It covers various metrics used for this purpose, including accuracy, precision, recall, F1 score, and ROC-AUC. The lessons build upon the dataset and model from the previous module, which involves predicting customer churn. The objective is to determine whether a customer is likely to leave or remain, using a logistic regression model.

Accuracy

Accuracy is a common metric used to evaluate classification models. It measures the proportion of correct predictions made by the model. However, accuracy alone may not provide a complete picture of the model’s performance, especially in imbalanced datasets.

Confusion matrix

The confusion matrix is a tabular representation of the model’s predictions against the actual values. It consists of four components: true positives, true negatives, false positives, and false negatives. These components are used to calculate other evaluation metrics such as precision, recall, and F1 score.

True positives (TP) are the cases where the model correctly predicts the positive class. True negatives (TN) are the cases where the model correctly predicts the negative class. False positives (FP) are the cases where the model incorrectly predicts the positive class. False negatives (FN) are the cases where the model incorrectly predicts the negative class.

Visually, the confusion matrix looks like this:

Using these components, the accuracy is nothing else than the sum of the true positives and true negatives divided by the total number of observations.

Accuracy = (TP + TN) / (TP + TN + FP + FN)

Precision

Precision measures the proportion of true positive predictions among all positive predictions made by the model. It is calculated as the ratio of true positives to the sum of true positives and false positives.

Precision = TP / (TP + FP)

Recall

Recall, also known as sensitivity or true positive rate, measures the proportion of true positive predictions among all actual positive cases. It is calculated as the ratio of true positives to the sum of true positives and false negatives.

Recall = TP / (TP + FN)

F1 score

The F1 score is the harmonic mean of precision and recall. It provides a balance between the two metrics and is useful when the classes are imbalanced.

An F1 score of 1 indicates a perfect model, while a score of 0 indicates a model that performs no better than random. The relative contribution of precision and recall to the F1 score are equal.

F1 Score = 2 * (Precision * Recall) / (Precision + Recall)

ROC curve

The Receiver Operating Characteristic (ROC) curve is a graphical representation of the trade-off between the true positive rate (recall), which we want to maximize, and the false positive rate, which we want to minimize, of a classification model. It is used to evaluate the performance of the model across different thresholds.

False positive rate (FPR) is the proportion of false positive predictions among all actual negative cases. It is calculated as the ratio of false positives to the sum of false positives and true negatives.

True positive rate (TPR) is the same as recall.

The ROC curve plots the TPR against the FPR for different threshold values.

Area Under the ROC curve

The Area Under the ROC Curve (AUC-ROC) is a metric used to evaluate the performance of a binary classification model. It measures the model’s ability to distinguish between the two classes (positive and negative) across all possible classification thresholds.

AUC-ROC answers the question, “What is the probability that the model will rank a randomly chosen positive instance higher than a randomly chosen negative instance?”

The AUC-ROC score ranges from 0 to 1, where a score of 0.5 indicates a model that performs no better than random, and a score of 1 indicates a perfect model.

Cross-validation

Cross-validation is a technique used to evaluate the performance of a machine learning model. It involves splitting the dataset into multiple subsets, training the model on some subsets, and testing it on others. This process is repeated multiple times to obtain a more reliable estimate of the model’s performance.

The most common form of cross-validation is k-fold cross-validation, where the dataset is divided into k subsets (folds). The model is trained on k-1 folds and tested on the remaining fold. This process is repeated k times, with each fold serving as the test set once.

This technique is applied in the parameter tuning process to select the best hyperparameters for the model.

Conclusion

This module provides an overview of the evaluation metrics used to assess the performance of classification models. It covers the importance of accuracy, precision, recall, F1-score, ROC-AUC, and the confusion matrix in evaluating the model’s performance.

The homework code can be found here.

This post is a recap of the Machine Learning Zoomcamp Module 3.

Below are the posts for the previous modules:

1. Machine Learning Zoomcamp Module 1 - points received: 9 (7/7 for questions + 2 bonus for learning in public)
2. Machine Learning Zoomcamp Module 2 - points received: 5 (5/6 for questions + 0 bonus for learning in public)

For module 2 I managed to score 5 points out of 6 for the questions. While the code and the result I got were correct, I interpreted the result incorrectly, choosing the wrong answer. Oh well, it happens…

The gist of the module

While the previous module was all about regression, i.e. predicting a continuous value, this module is about classification, i.e. predicting a category. As we have just two categories, we are talking about binary classification.

The dataset

As with all machine learning problems, we begin with the dataset. In this module, the dataset focuses on predicting customer churn, determining whether a customer is likely to leave or stay. The dataset used for the homework focuses on predicting if a customer has subscribed a term deposit or not.

Data preparation and EDA

One of the first steps in implementing a machine learning model is to prepare the data and perform exploratory data analysis (EDA).

Regarding data preparation, we need to:

• bring the columns and values within to a common format, i.e. lowercase, no spaces, etc.
• perform some numerical conversions
• handling missing values
• encoding categorical variables, i.e. yes/no to 1/0

Regarding EDA, we need to:

• check the distribution of the target variable
• check the distribution of the features
• calculate the correlation between the features
• calculate the mutual information between the features and the target variable

Training a model

The next step is to train a model. In this module, we used the logistic regression model. The logistic regression model is a linear model that predicts the probability of an outcome. In our case the outcome is binary.

Before proceeding with the training, we need to split the data into training, validation, and test sets. The training set is used to train the model, the validation set is used to tune the hyperparameters, and the test set is used to evaluate the model.

from sklearn.model_selection import train_test_split

df_train, df_remaining = train_test_split(df, test_size=0.4, random_state=42)

df_val, df_test = train_test_split(df_remaining, test_size=0.5, random_state=42)

y_train = df_train.y.values
y_val = df_val.y.values
y_test = df_test.y.values

Performing one-hot encoding

Before training the model, we need to perform one-hot encoding on the categorical variables. One-hot encoding is a process of converting categorical variables into a form that could be provided to ML algorithms to do a better job in prediction. Below we use the homework dataset to perform one-hot encoding.

from sklearn.feature_extraction import DictVectorizer

numerical_features = ['balance', 'previous', 'pdays', 'campaign', 'age', 'day', 'duration']

categorical_features = ['education', 'housing', 'month', 'contact', 'marital', 'poutcome', 'job']

dict_train = df_train[numerical_features+categorical_features].to_dict(orient='records')

dv = DictVectorizer(sparse=False)
X_train = dv.fit_transform(dict_train)

After the one-hot encoding the feature names are stored in the dv.get_feature_names_out() method, and look like this:

[
  'age',
  'balance',
  'campaign',
  'contact=cellular',
  'contact=telephone',
  'contact=unknown',
  'day',
  'duration',
  'education=primary',
  'education=secondary',
  'education=tertiary',
  'education=unknown',
  'housing=no',
  ...
  'housing=yes',
  'poutcome=failure',
  'poutcome=other',
  'poutcome=success',
  'poutcome=unknown',
  'previous'
]

Training a logistic regression model

from sklearn.linear_model import LogisticRegression

model = LogisticRegression(solver='liblinear', C=1.0, max_iter=1000, random_state=42)
model.fit(X_train, y_train)

The model is trained using the training set and the target variable. The solver parameter is set to liblinear, which is a good choice for small datasets. The C parameter is set to 1.0, which is the default value. The max_iter parameter is set to 1000, which is the maximum number of iterations taken for the solvers to converge. The random_state parameter is set to 42 to ensure reproducibility.

Making predictions

After training the model, we can make predictions using the validation set.

dict_val = df_val.to_dict(orient='records')
dv = DictVectorizer(sparse=False)
X_val = dv.fit_transform(dict_val)
y_pred = model.predict_proba(X_val)[:, 1]

One last step is to make a decission based on the predictions. In this case, we use a threshold of 0.5 to determine if a customer has subscribed a term deposit or not.

y_decission = y_pred >= 0.5

After this step, we can calculate the accuracy of the model.

(y_val == y_decission).mean()

We can do one final check by calculating the accuracy of the model using the test set.

dict_test = df_test.to_dict(orient='records')
X_test = dv.fit_transform(dict_test)
y_pred = model.predict_proba(X_test)[:, 1]
y_decission = y_pred >= 0.5
(y_test == y_decission).mean()

Conclusion

This module was a great introduction to binary classification. We learned how to prepare the data, perform EDA, train a model, and make predictions. We also learned how to evaluate the model using the accuracy metric.

The homework code can be found here.

In this post I will make a recap of the Machine Learning Zoomcamp Module 2.

The posts for the previous modules, together with the points I received for each module’s homework, are listed below:

1. Machine Learning Zoomcamp Module 1 - points received: 9 (7/7 for questions + 2 bonus for learning in public)

The gist of the module

The problem tackled in this module was the prediction of car prices based on a dataset containing car features.

This type of problem is called a regression problem, and the main idea is to try to predict a continuous value, in this case, the price of a car, based on a set of car features, using a Machine Learning model. For this problem, the model used was the Linear Regression model.

To get a real taste of how such a problem could be solved, the model was implemented from scratch using Python and NumPy, while the dataset was explored and manipulated using Pandas and Matplotlib.

The dataset

As with all things in Machine Learning, the first step is to get the data and make a sense of it.

The dataset used in this module was the Car Features and MSRP from Kaggle.

Note that I will be using another dataset, the one from the homework, to illustrate the concepts discussed in this post. While the datasets are different, the process is the same.

As it is often the case, the dataset may require some cleaning and preprocessing before it could be used to train a model. One such action was normalizing the column names. To get rid of inconsistencies, all columns were converted to lowercase, and the spaces were replaced with underscores.

url = 'https://raw.githubusercontent.com/alexeygrigorev/datasets/master/laptops.csv'

df = pd.read_csv(url)

cols = df.columns.str.lower().str.replace(r'\s+','_', regex=True)
df.columns = cols

A similar conversion process can be applied to the values in the columns, to make sure that the data is consistent and easy to work with.

string_columns = list(df.dtypes[df.dtypes == 'object'].index)
for col in string_columns:
    df[col] = df[col].str.lower().str.replace(r'\s+','_', regex=True)

Another common preprocessing step is to handle missing values. For instance, if a column contains missing values, one could replace them with the mean of the column.

df.screen = df.screen.fillna(df.screen.mean())

Looking at the target variable, i.e. the variable we want to predict, is also important. In this case, the target variable is the price of the car. It is a good idea to check the distribution of the target variable, as it can give us some insights into the data. One way to do this is to plot a histogram of the target variable, using a library like Seaborn.

import seaborn as sns
fp = df.final_price
sns.histplot(fp, bins=50)

One imediate observation could be that the data is not normally distributed. This could be a problem, as the Linear Regression model works better with normally distributed data. One way to address this issue is to apply a transformation to the target variable. One such transformation is the logarithmic transformation.

fp = np.log1p(df.final_price)

Plotting the histogram of the transformed target variable could show a more normally distributed data.

Before training the model, we have to split the data into training, validation and testing sets. The training set is used to train the model, the validation set is used to tune the hyperparameters of the model, and the testing set is used to evaluate the model’s performance. A common split is 60% for training, 20% for validation, and 20% for testing.

Without relying on libraries like Scikit-Learn, we can split the data by shuffling the indices of the dataset and then splitting the indices into the three sets.

Training the model

Training the model refers to the process by which an algorithm learns the relationship between the input features and the target variable. In this case, the algorithm used was the Linear Regression model. The artifacts of the training process are the weights and the bias of the model.

Linear Regression is a simple model that tries to find the best line that fits the data. The line is defined by the equation y = w1*x1 + w2*x2 + ... + wn*xn + b, where w1, w2, ..., wn are the weights, x1, x2, ..., xn are the input features, and b is the bias. In the module bias was named w0. It is also called the intercept, which is the value of the target variable when all the input features are zero, i.e. the value of the target variable when there is no influence from the input features.

For a single feature, x, and its weight, w, the equation becomes y = w*x + b. This is nothing else than the equation of a line, where w is the slope of the line, and b is the y-intercept. The intercept is where the line crosses the y-axis, and the slope is the rate at which the line rises or falls, or mathematically, the change in y divided by the change in x.

Considering we have many observations, we can arrange the input features in a matrix, X, and the target variable in a vector, y. The equation of the model becomes y = X*w + b, where X is a matrix of shape (n_samples, n_features), w is a vector of shape (n_features, 1), and b is a scalar.

Now we are talking about matrix-vector multiplication, which means we can think at b as a vector of ones, and add it as a column to the matrix X. This way, the equation becomes y = X*w, where X is a matrix of shape (n_samples, n_features + 1), w is a vector of shape (n_features + 1, 1), and y is a vector of shape (n_samples, 1).

If the matrix X is invertible (i.e. it is a square matrix and its determinant is not zero), we can find the weights, w, by multiplying the inverse of X with y.

If matrix X is not invertible, we have to make use of the transpose of X, X^T, which multiplied with X gives a square matrix, X^T*X, also called the Gram matrix, which could be invertible. If this is the case, the weights can be found as w = (X^T*X)^-1*X^T*y, where ^-1 is the matrix inverse.

If the Gram matrix is not invertible, we need to talk about regularization, a bit later on in this post.

Feature engineering

In order to improve the model performance, we can engineer the features. This means that we can create new features from the existing ones, or transform the existing features in a way that makes them more informative. This basically means that the feature matrix, X, can contain more columns than the original dataset.

For example, if one of the features is the status of a laptop, i.e. weather it is new or refurbished, we can add two new columns to the feature matrix, one for each status, let’s say, is_new and is_refurbished. If the laptop is new, the value of is_new is 1, and the value of is_refurbished is 0. If the laptop is refurbished, the value of is_new is 0, and the value of is_refurbished is 1.

Regularization

We saw that the Gram matrix may not be invertible. This happens when the matrix is singular, i.e. it has a determinant of zero. This could happen when the number of features is not the same as the number of observations, or when the features are collinear, i.e. they are linearly dependent. Collinearity just means that one or more columns in the feature matrix can be expressed as a linear combination of the other columns. For instance, if we have two columns, x1 and x2, and x2 = 2*x1, then the two columns are collinear.

One way to address this issue is to add a factor to the diagonal of the Gram matrix, called the regularization term. This term is a scalar, r, multiplied with the identity matrix, I, and added to the Gram matrix. The equation becomes w = (X^T*X + r*I)^-1*X^T*y.

This equation can be nicely expressed in Python code:

def train_linear_regression_reg(X, y, r=0.0):
  ones = np.ones(X.shape[0]) # create a vector of ones
  X = np.column_stack([ones, X]) # add the vector of ones as a column to the matrix X

  XTX = X.T.dot(X) # calculate the Gram matrix
  reg = r * np.eye(XTX.shape[0]) # create the regularization term
  XTX = XTX + reg # add the regularization term to the Gram matrix

  XTX_inv = np.linalg.inv(XTX) # calculate the inverse of the Gram matrix
  w = XTX_inv.dot(X.T).dot(y) # calculate the weights

  return w[0], w[1:] # return the bias and the weights

Model evaluation

Once we have our model trained, i.e. we know the weights and the bias, we can evaluate its performance. One way to do this is to calculate the root mean squared error (RMSE) of the model. The RMSE is a measure of the difference between the predicted values and the actual values. The lower the RMSE, the better the model.

The RMSE is calculated as the square root of the mean of the squared differences between the predicted values and the actual values. The equation is RMSE = sqrt(mean((y_pred - y)^2)).

def rmse(y, y_pred):
  se = (y - y_pred) ** 2
  mse = se.mean()
  return np.sqrt(mse)

We can do this for the validation set and the testing set, and compare the RMSE values. If the RMSE of the testing set is much higher than the RMSE of the validation set, it could be a sign of overfitting, i.e. the model is too complex and it is fitting the noise in the data, rather than the underlying pattern.

Once we have a fairly good model, we can use it to make predictions on new data.

Using the model

Using the model just means to apply the weights and the bias to the new data. The equation is the same as the one used to train the model, y = X*w + b, where X is the feature matrix of the new data, w is the weights of the model, and b is the bias.

def predict(X, w, b):
  return X.dot(w) + b

Conclusion

In this module, we learned how to train a Linear Regression model from scratch, using Python and NumPy. We also learned how to preprocess the data, engineer the features, and evaluate the model. We also learned about regularization and how to address the issue of collinearity.

The homework code can be found here.

In one of my previous posts, I mentioned that I was taking the Machine Learning Zoomcamp course. I have completed the first module of the course, and I wanted to share a recap of what I learned. In addition to that, I’d go one step further and add some extra bits of information here and there.

What is Machine Learning?

To answer this, we need to understand what learning is. Learning is the process of acquiring knowledge or skills through experience, study, or being taught. A simple Google search would give you this definition. Wikipedia defines learning similarly, adding that “[…]the ability to learn is possessed by humans, non-human animals, and some machines[…]”. When learning is done by machines, we refer to it as Machine Learning (ML). In simple terms, ML is the process of teaching a computer to learn from data. The computer uses the data to identify patterns and make decisions without being explicitly programmed to do so.

Why do we need ML?

Why would we want a machine to learn? Why is a rule-based system not enough?

The answer is two-fold. First, we cannot anticipate all possible future situations. Think of a robot navigating a maze, for instance. Second, we sometimes just don’t know how to implement the solution. For example, consider the task of recognizing handwritten digits. Without ML algorithms, this would be a very challenging task to implement.

Recognizing people’s faces in images is another good example, we as humans can do this task effortlessly, but writing a computer program to do the same is not trivial without ML.

A rule-based system would require humans to write down all the rules for the computer to follow, whereas with ML, the computer can learn the rules from the data. Let’s consider two other examples, spam detection and car price prediction. Applying a rule-based system to these tasks would be very difficult. In the case of spam detection, the rules would be constantly changing as spammers come up with new ways to bypass the system. In the case of car price prediction, there are so many factors that can affect the price of a car that it would be impossible to write down all the rules. With ML, the computer can learn the rules from the data and make predictions based on that.

How does ML work?

To put it simply, ML works by learning from examples. The computer is given a set of examples, and it uses these examples to build a model, which is then used to make predictions on new data. The set of examples is called the training data, and the process of building the model is called training. The model is a mathematical representation of the patterns in the data.

Types of ML

There are three main types of ML: supervised learning, unsupervised learning, and reinforcement learning.

This post will touch upon supervised learning, while the other two types will be covered in future posts.

Supervised ML (SML)

In supervised learning, the computer observes input-output pairs and learns a function that maps new inputs to outputs. The input is called the feature, while the output is called the label.

For example, if we are trying to predict the price of a house based on its size, the size of the house would be the input or the feature, and the price would be the output or the label. Of course, we would need a lot more features than just the size of the house to make an accurate prediction.

We put all the features together in a feature matrix, usually called X, where each row represents a different example, and each column represents a different feature. The labels are put in a target vector, usually called y, for each example in the feature matrix, there is a corresponding label in the target vector. The model is then a function that takes X as input and tries to predict values as close as possible to y. Formally, we can write this as f(X) ≈ y.

If matrix, vector, or function are not familiar terms to you, this is where maths comes in, but to keep things simple, you can think of a matrix as a table, a vector as a list of numbers, and a function as a rule that takes an input and gives an output.

Let’s see some types of supervised learning problems:

• Regression: The output is a continuous value (the price of a house)
• Classification: The output is a discrete value (spam or not spam / ham)
• Ranking: The output is a ranking of items (ranking search results)

It’s all about the data

In ML, data is everything. The quality of the data is crucial to the success of the model. If the data is bad, the model will be bad. If the data is good, the model will be good, given no other errors have been made along the way.

You might ask, is there a process to follow when working with data? The answer is yes, there is a process called the CRISP-DM process, which stands for Cross-Industry Standard Process for Data Mining. The process consists of six phases:

• Business Understanding, i.e. understanding the problem you are trying to solve
• Data Understanding, i.e. understanding the data you have
• Data Preparation, i.e. preparing the data for modeling
• Modeling, i.e. building the model
• Evaluation, i.e. evaluating the model
• Deployment, i.e. deploying the model

The process is iterative, meaning that you might have to go back and forth between the phases until you get the desired results.

Fortunately, there are tools and libraries that can help you with each phase of the process. Following posts will cover some of these tools and libraries.

Choosing a model

There are many types of models used in machine learning, each suitable for different tasks like classification, regression, clustering, and more. Here are some commonly used machine learning models:

• Linear Regression
• Logistic Regression
• Decision Trees
• Random Forests
• Support Vector Machines
• Neural Networks
• Convolutional Neural Networks
• Recurrent Neural Networks
• K-Nearest Neighbors
• K-Means Clustering
• Principal Component Analysis
• Gradient Boosting
• Naive Bayes
• etc.

Considering the vast number of models available, how do you choose the right one for your task? The answer is, it depends. It depends on the problem you are trying to solve, the data you have, and the resources you have available. In general, it is a good idea to start with a simple model and then move on to more complex models if needed.

One way to choose a model is to try different models and see which one performs best on your data. This process is called model selection.

For this we need to divide our data into multiple sets.

One approach is to devide it into two sets: the training set and the test set. The training set is used to train the model, while the test set is used to evaluate the model.

Another approach is to divide the data into three sets: the training set, the validation set, and the test set.

The training set is used to train the model, the validation set is used to evaluate the model, this might imply additional tweaking of the model, and the test set is used to evaluate the final model. The training-evaluation-tweaking process is repeated and the best model is chosen based on the performance on the validation set. This model is then evaluated on the test set.

Tools and Libraries

Previously I mentioned that there are tools and libraries that can help you with each phase of the CRISP-DM process. Two of the most commonly used libraries in Python are NumPy and Pandas. Pandas is used for data manipulation and analysis, while NumPy is used for numerical computing.

See NumPy and Pandas for more information.

To make use of these tools one has to set up their Python environment. One way to do this is to use Anaconda, which is a distribution of Python that comes with all the tools and libraries you need for data science and machine learning. Once you have Anaconda installed, you can optionally create a virtual environment, activate it, install the libraries you need, and start working on your project.

conda create -n myenvironment python=3.11
conda activate myenvironment
conda install numpy pandas scikit-learn seaborn jupyter

If running all this locally is not possible for you, you can use Google Colab, which is a free cloud-based service that allows you to run Python code in the browser, or even set up everything yourself using some cloud service like AWS, Azure, or GCP.

Conclusion

In this post, I covered the basics of machine learning, the types of machine learning, the importance of data, the CRISP-DM process, choosing a model, and some tools and libraries that can help you with machine learning.

Pandas is a fast, powerful, flexible and easy to use open source data analysis and manipulation tool, built on top of the Python programming language. It was created by Wes McKinney in 2008.

The name has nothing to do with the cute animal, it’s derived from the term “panel data”, an econometrics term for data sets that include observations over multiple time periods for the same individuals. It is also a play on the phrase “Python data analysis”.

A good resource to learn more about Pandas is the book “Python for Data Analysis” by Wes McKinney.

The environment

Make sure you can run the following commands in your terminal, without errors:

python3.12 --version
Python 3.12.4

Inside the Python’s REPL (Read-Eval-Print Loop), you get inside by typing python3.12 in the terminal:

import numpy as np
import pandas as pd
print(np.__version__)
1.26.4
print(pd.__version__)
2.2.2

If you get similar results, you are ready to go.

Data Structures

The main data structures in Pandas are the Series and DataFrame objects.

The Series object is a one-dimensional array-like object containing a sequence of values of the same type and an associated array of data labels, called its index.

The DataFrame object is a two-dimensional array with both row and column indices. The name comes from the R data.frame object.

Operations on Series

Below are some examples of using the Series object.

Create a Series with a default index

The string representation of a Series shows the index on the left and the values on the right.

s = pd.Series([1,2,3,4,5,6,7])
s
0    1
1    2
2    3
3    4
4    5
5    6
6    7
dtype: int64

s.array
<NumpyExtensionArray>
[1, 2, 3, 4, 5, 6, 7]
Length: 7, dtype: int64

s.index
RangeIndex(start=0, stop=7, step=1)

Create a Series with a custom index

s = pd.Series([7,77,777], index= ['a', 'b', 'c'])
s
a      7
b     77
c    777
dtype: int64

Selecting values from a Series

s = pd.Series([7,77,777], index= ['a', 'b', 'c'])
s['b']
77
s[['c','b','a']]
c    777
b     77
a      7
dtype: int64

From Python dictionary to Series and back

data = {1:'a', 2:'b', 3:'c'}
s = pd.Series(data)
s
1    a
2    b
3    c
dtype: object

s.to_dict()
{1: 'a', 2: 'b', 3: 'c'}

# passing `index` changes the order of the elements
s = pd.Series(data, index=[3,2,1])
s
3    c
2    b
1    a
dtype: object

Data alignment for arithmetic operations

Exemplified for addition, but it works for other operations as well.

s1 = pd.Series({'a':1, 'b':2})
s2 = pd.Series({'a':9, 'b':8})
s1 + s2
a    10
b    10
dtype: int64

Operations on DataFrame

Creating a DataFrame from a dictionary of equal-length lists

frame = pd.DataFrame({'even':[0,2,4,6,8], 'odd':[1,3,5,7,9], 'prime':[1,2,3,5,7]})
frame
   even  odd  prime
0     0    1      1
1     2    3      2
2     4    5      3
3     6    7      5
4     8    9      7

# to change the order of the columns use the `columns` parameter
frame = pd.DataFrame({'even':[0,2,4,6,8], 'odd':[1,3,5,7,9], 'prime':[1,2,3,5,7]}, columns=['prime','odd','even'])
frame
   prime  odd  even
0      1    1     0
1      2    3     2
2      3    5     4
3      5    7     6
4      7    9     8

Retrieving a column from a DataFrame as a Series

frame = pd.DataFrame({'even':[0,2,4,6,8], 'odd':[1,3,5,7,9], 'prime':[1,2,3,5,7]})
primes = frame['prime']
primes
0    1
1    2
2    3
3    5
4    7
Name: prime, dtype: int64
type(primes)
<class 'pandas.core.series.Series'>

Retrieving a row from a DataFrame as a Series

frame = pd.DataFrame({'even':[0,2,4,6,8], 'odd':[1,3,5,7,9], 'prime':[1,2,3,5,7]})
frame.loc[4]
even     8
odd      9
prime    7
Name: 4, dtype: int64

frame.iloc[4]
even     8
odd      9
prime    7
Name: 4, dtype: int64

Assigning values to a column

frame
   even  odd  prime
0     0    1      1
1     2    3      2
2     4    5      3
3     6    7      5
4     8    9      7
frame.even = [2,4,6,8,10]
frame
   even  odd  prime
0     2    1      1
1     4    3      2
2     6    5      3
3     8    7      5
4    10    9      7

Adding a new column

frame
   even  odd  prime
0     0    1      1
1     2    3      2
2     4    5      3
3     6    7      5
4     8    9      7
frame['prime_is_odd'] = frame['prime'] % 2 != 0
frame
   even  odd  prime  prime_is_odd
0     0    1      1          True
1     2    3      2         False
2     4    5      3          True
3     6    7      5          True
4     8    9      7          True

Deleting a column

frame.columns
Index(['even', 'odd', 'prime', 'prime_is_odd'], dtype='object')

del frame['prime_is_odd']

frame.columns
Index(['prime', 'odd', 'even'], dtype='object')

Transposing a DataFrame

i.e. swaping rows and columns

frame
   even  odd  prime
0     0    1      1
1     2    3      2
2     4    5      3
3     6    7      5
4     8    9      7
frame.T
       0  1  2  3  4
even   0  2  4  6  8
odd    1  3  5  7  9
prime  1  2  3  5  7

From DataFrame to Numpy’s ndarray

NumPy arrays will be covered in another post. For now you can think of them as a more powerful version of Python’s lists.

frame
   even  odd  prime
0     0    1      1
1     2    3      2
2     4    5      3
3     6    7      5
4     8    9      7
frame.to_numpy()
array([[0, 1, 1],
       [2, 3, 2],
       [4, 5, 3],
       [6, 7, 5],
       [8, 9, 7]])

The Index object

• is immutable
• can be shared between data structures
• is array-like and also behaves like a fixed-size set
• can contain duplicate labels

index = pd.Index(['a','a','b','c'])
index
Index(['a', 'a', 'b', 'c'], dtype='object')

index[1:]
Index(['a', 'b', 'c'], dtype='object')

# index[0] = 'x' -> TypeError: Index does not support mutable operations

s = pd.Series([1,1,2,3], index=index)
s
a    1
a    1
b    2
c    3
dtype: int64

s.index is index 
True

frame = pd.DataFrame({'even':[0,2,4,6,8], 'odd':[1,3,5,7,9], 'prime':[1,2,3,5,7]})
frame.columns
Index(['even', 'odd', 'prime'], dtype='object')

frame.index
RangeIndex(start=0, stop=5, step=1)

'odd' in frame.columns
True

3 in frame.index
True

Essential functionality

Reindexing

i.e. creates a new object with the values rearranged according to the new index.

obj = pd.Series([1,2,3], index=['a', 'b', 'c'])
obj
a    1
b    2
c    3
dtype: int64
obj.reindex(['c','b','a'])
c    3
b    2
a    1
dtype: int64

Dropping entries from an axis

of a Series:

obj = pd.Series([1,2,3], index=['a', 'b', 'c'])
obj.drop('b')
a    1
c    3
dtype: int64

of a DataFrame:

This line np.arange(4).reshape((2,2)) creates a 2x2 matrix with values from 0 to 3.

frame = pd.DataFrame(np.arange(4).reshape((2,2)), index=['a', 'b'], columns=['c', 'd'])
frame
   c  d
a  0  1
b  2  3

frame.drop(index=['b'], columns=['d'])
   c
a  0

Indexing, selection, and filtering

obj = pd.Series([1,2,3,4], index=['a', 'b', 'c', 'd'])
obj['b']
2
obj.loc[['b','a','d']]
b    2
a    1
d    4
dtype: int64
obj[1] # Future warning: integer keys will always be treated as labels
2
obj.iloc[1]
2
obj[2:4]
c    3
d    4
dtype: int64
obj[['b','a','d']]
b    2
a    1
d    4
dtype: int64
obj[obj > 2]
c    3
d    4
dtype: int64

frame = pd.DataFrame(np.arange(4).reshape((2,2)), index=['a', 'b'], columns=['c', 'd'])
frame
   c  d
a  0  1
b  2  3
frame.loc['b']
c    2
d    3
Name: b, dtype: int64

frame.iloc[1]
c    2
d    3
Name: b, dtype: int64

Sorting

obj = pd.Series([4,3,2,1], index=['d', 'c', 'b', 'a'])
obj.sort_index()
a    1
b    2
c    3
d    4
dtype: int64

frame = pd.DataFrame(np.arange(4).reshape((2,2)), index=['b', 'a'], columns=['d', 'c'])
frame
   d  c
b  0  1
a  2  3
frame.sort_index()
   d  c
a  2  3
b  0  1

frame.sort_index(axis='columns')
   c  d
b  1  0
a  3  2

frame.sort_index(axis='columns', ascending=False)
   d  c
b  0  1
a  2  3

Reading and writing data

Pandas can read and write data from and to a variety of formats, such as CSV, JSON, Excel, SQL databases, and more.

In this example, we will be focusing on reading and writing CSV files.

Reading a CSV file

cat data.csv
col1,col2,col3
1,2,foo
3,4,bar
5,6,baz

pd.read_csv('data.csv')
   col1  col2 col3
0     1     2  foo
1     3     4  bar
2     5     6  baz

In case the file does not have a header row, read_csv, has some parameters to help with that.

cat data_no_header.csv
1,2,foo
3,4,bar
5,6,baz

pd.read_csv('data_no_header.csv', header=None)
   0  1    2
0  1  2  foo
1  3  4  bar
2  5  6  baz

pd.read_csv('data_no_header.csv', names=['c1','c2','c3'])
   c1  c2   c3
0   1   2  foo
1   3   4  bar
2   5   6  baz

We can also specify the index column to be used instead of the default one.

pd.read_csv('data.csv')
   col1  col2 col3
0     1     2  foo
1     3     4  bar
2     5     6  baz
pd.read_csv('data.csv', index_col='col3')
      col1  col2
col3
foo      1     2
bar      3     4
baz      5     6

In case the csv file uses another separator, we can specify it using the sep parameter.

pd.read_csv('data_with_spaces.csv', sep='\\s+')
   col1  col2 col3
0     1     2  foo
1     3     4  bar
2     5     6  baz

read_csv also allows us to skip specific rows, or to only read a specific number of rows.

pd.read_csv('data_with_spaces.csv', sep='\\s+')
   col1  col2 col3
0     1     2  foo
1     3     4  bar
2     5     6  baz
pd.read_csv('data_with_spaces.csv', sep='\\s+', skiprows=[1,2])
   col1  col2 col3
0     5     6  baz

pd.read_csv('data_with_spaces.csv', sep='\\s+')
   col1  col2 col3
0     1     2  foo
1     3     4  bar
2     5     6  baz
3     7     8  bax
4     9    10  bay
pd.read_csv('data_with_spaces.csv', sep='\\s+', nrows=3)
   col1  col2 col3
0     1     2  foo
1     3     4  bar
2     5     6  baz

Writing a CSV file

to_csv writes a DataFrame to a CSV file. It offers a variety of parameters to customize the output, among them are sep, header, and columns.

data = pd.read_csv('data_with_spaces.csv', sep='\\s+', nrows=3)
data
   col1  col2 col3
0     1     2  foo
1     3     4  bar
2     5     6  baz
data.to_csv('new_data.csv', sep='|', header=True, columns=['col2','col3'])

cat new_data.csv
|col2|col3
0|2|foo
1|4|bar
2|6|baz

Data cleaning

Missing data

Python’s None and NumPy’s np.nan(np comes from import numpy as np) are considered missing data in Pandas.

na stands for “not available”.

s = pd.Series([1,2,np.nan,None,3,4])
s
0    1.0
1    2.0
2    NaN
3    NaN
4    3.0
5    4.0
dtype: float64

s.isna()
0    False
1    False
2     True
3     True
4    False
5    False
dtype: bool

s.notna()
0     True
1     True
2    False
3    False
4     True
5     True
dtype: bool