Build clickhouse remote development environment with vscode(v24.8.11.5-lts)

1 Build dev docker image FROM docker.io/ubuntu:22.04 RUN rm /bin/sh && ln -s /bin/bash /bin/sh RUN apt-get -y update RUN apt-get install -y curl vim git ssh openssh-server cmake ccache python3 ninja-build nasm yasm gawk lsb-release wget software-properties-common gnupg RUN ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa RUN touch ~/.ssh/authorized_keys RUN bash -c "$(wget -O - https://apt.llvm.org/llvm.sh)" RUN echo "export CC=clang-18" >> /root/.bashrc RUN echo "export CXX=clang++-18" >> /root/.bashrc RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y RUN echo "export PATH=~/.cargo/bin:$PATH" >> /root/.bashrc RUN /root/.cargo/bin/rustup toolchain install nightly-2024-12-01 RUN /root/.cargo/bin/rustup default nightly-2024-12-01 RUN /root/.cargo/bin/rustup component add rust-src The rustup installation may fail due to network problems. Just try again, or write an automatic retry script. After docker build, you will get a development image, named xxx-clickhouse-dev-env:24.8 in this article. 2 Download clickhouse code git clone git@github.com:ClickHouse/ClickHouse.git /clickhouse-24 cd /clickhouse-24 git checkout v24.8.11.5-lts Download all submodules #!/bin/sh while : do git submodule update --init --recursive --force sleep 1 done The loop is because the code clone is unstable and may fail. When the actual clone does not appear in the log, manually kill the above process. 3 Set up development environment remote development environment docker run -itd --name xxx -v /clickhouse-24:/data/clickhouse --privileged=true -p xxx:22 --cap-add="NET_ADMIN" --security-opt seccomp=unconfined xxx-clickhouse-dev-env-24.8 bash Execute the following command in the docker container to enable ssh. echo xxx >> ~/.ssh/authorized_keys # Write the token in your local id_rsa.pub /etc/init.d/ssh start local development environment Download vscode from https://code.visualstudio.com/. Install extension remote-SSH next, then we can connect to remote server in vscode. Add a new ssh host in configuration like below. Host clickhouse-24 HostName User root Port Then you can connect to the server as clickhouse-24, then open the code directory /data/clickhouse. Next we need to install necessary extensions on remote server. Compile Add .vscode directory in the working directory and a tasks.json file with the following contents: { "version": "2.0.0", "tasks": [ { "type": "shell", "label": "cmake", "group": "build", "command": "cmake -DCMAKE_BUILD_TYPE=Debug -DENABLE_CCACHE=1 -DCMAKE_C_COMPILER=/usr/lib/llvm-18/bin/clang -DCMAKE_CXX_COMPILER=/usr/lib/llvm-18/bin/clang++ -DCMAKE_PREFIX_PATH=/usr/lib/llvm-18/ -DENABLE_JEMALLOC=ON -DENABLE_TESTS=OFF -DCOMPILER_FLAGS=-DNDEBUG -DWERROR=OFF -G Ninja -B build", "options": { "cwd": "${workspaceFolder}" }, "problemMatcher": [], "presentation": { "echo": true, "reveal": "always", "focus": true, "panel": "shared", "showReuseMessage": true, "clear": false } }, { "type": "shell", "label": "ninja clickhouse", "group": "build", "command": "ninja clickhouse clickhouse-server clickhouse-client -j16", "options": { "cwd": "${workspaceFolder}/build" } // "dependsOn": "cmake", }, { "type": "shell", "label": "ninja all", "group": "build", "command": "ninja -j16", "options": { "cwd": "${workspaceFolder}/build" } // "dependsOn": "cmake", } ] } Do cmake first and then ninja clickhouse, ninja clickhouse will cost several hours. Add settings.json file in dir .vscode to enable code reference. { "clangd.path": "/usr/lib/llvm-18/bin/clangd", "clangd.checkUpdates": false, "clangd.arguments": [ "--background-index", "--compile-commands-dir=build", "-j=12", "--query-driver=/usr/lib/llvm-18/bin/clang++", "--clang-tidy", "--clang-tidy-checks=performance-,bugprone-", "--all-scopes-completion", "--completion-style=detailed", "--header-insertion=iwyu", "--pch-storage=disk" ], "clangd.onConfigChanged": "restart", "lldb.commandCompletions": true, "lldb.dereferencePointers": true, "lldb.evaluateForHovers": true, "lldb.launch.expressions": "simple", "lldb.showDisassembly": "never", "lldb.verboseLogging": true, // cpp_tools Config "C_Cpp.autocomplete": "Disabled", "C_Cpp.formatting": "Disabled", "C_Cpp.errorSquiggles": "Disabled", "C_Cpp.intelliSenseEngine"

Jan 16, 2025 - 12:34

Build clickhouse remote development environment with vscode(v24.8.11.5-lts)

1 Build dev docker image

FROM docker.io/ubuntu:22.04
RUN rm /bin/sh && ln -s /bin/bash /bin/sh
RUN apt-get -y update
RUN apt-get install -y curl vim git ssh openssh-server cmake ccache python3 ninja-build nasm yasm gawk lsb-release wget software-properties-common gnupg
RUN ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
RUN touch ~/.ssh/authorized_keys
RUN bash -c "$(wget -O - https://apt.llvm.org/llvm.sh)"
RUN echo "export CC=clang-18" >> /root/.bashrc
RUN echo "export CXX=clang++-18" >> /root/.bashrc
RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
RUN echo "export PATH=~/.cargo/bin:$PATH" >> /root/.bashrc
RUN /root/.cargo/bin/rustup toolchain install nightly-2024-12-01
RUN /root/.cargo/bin/rustup default nightly-2024-12-01
RUN /root/.cargo/bin/rustup component add rust-src

The rustup installation may fail due to network problems. Just try again, or write an automatic retry script.

After docker build, you will get a development image, named xxx-clickhouse-dev-env:24.8 in this article.

2 Download clickhouse code

git clone git@github.com:ClickHouse/ClickHouse.git /clickhouse-24
cd /clickhouse-24
git checkout v24.8.11.5-lts

Download all submodules

#!/bin/sh
while :
do
git submodule update --init --recursive --force
sleep 1
done

The loop is because the code clone is unstable and may fail.
When the actual clone does not appear in the log, manually kill the above process.

3 Set up development environment

remote development environment

docker run -itd --name xxx -v /clickhouse-24:/data/clickhouse --privileged=true -p xxx:22 --cap-add="NET_ADMIN" --security-opt seccomp=unconfined xxx-clickhouse-dev-env-24.8 bash

Execute the following command in the docker container to enable ssh.

echo xxx >> ~/.ssh/authorized_keys # Write the token in your local id_rsa.pub
/etc/init.d/ssh start

local development environment

Download vscode from https://code.visualstudio.com/.
Install extension remote-SSH next, then we can connect to remote server in vscode.

Add a new ssh host in configuration like below.

Host clickhouse-24
    HostName 
    User root
    Port

Then you can connect to the server as clickhouse-24, then open the code directory /data/clickhouse.

Next we need to install necessary extensions on remote server.

Compile

Add .vscode directory in the working directory and a tasks.json file with the following contents:

{
    "version": "2.0.0",
    "tasks": [
        {
            "type": "shell",
            "label": "cmake",
            "group": "build",
            "command": "cmake -DCMAKE_BUILD_TYPE=Debug -DENABLE_CCACHE=1 -DCMAKE_C_COMPILER=/usr/lib/llvm-18/bin/clang -DCMAKE_CXX_COMPILER=/usr/lib/llvm-18/bin/clang++ -DCMAKE_PREFIX_PATH=/usr/lib/llvm-18/ -DENABLE_JEMALLOC=ON -DENABLE_TESTS=OFF -DCOMPILER_FLAGS=-DNDEBUG -DWERROR=OFF -G Ninja -B build",
            "options": {
                "cwd": "${workspaceFolder}"
            },
            "problemMatcher": [],
            "presentation": {
                "echo": true,
                "reveal": "always",
                "focus": true,
                "panel": "shared",
                "showReuseMessage": true,
                "clear": false
            }
        },
        {
            "type": "shell",
            "label": "ninja clickhouse",
            "group": "build",
            "command": "ninja clickhouse clickhouse-server clickhouse-client -j16",
            "options": {
                "cwd": "${workspaceFolder}/build"
            }
            // "dependsOn": "cmake",
            },
        {
            "type": "shell",
            "label": "ninja all",
            "group": "build",
            "command": "ninja -j16",
            "options": {
            "cwd": "${workspaceFolder}/build"
            }
            // "dependsOn": "cmake",
        }
    ]
}

Do cmake first and then ninja clickhouse, ninja clickhouse will cost several hours.

Add settings.json file in dir .vscode to enable code reference.

{
    "clangd.path": "/usr/lib/llvm-18/bin/clangd",
    "clangd.checkUpdates": false,
    "clangd.arguments": [
        "--background-index",
        "--compile-commands-dir=build",
        "-j=12",
        "--query-driver=/usr/lib/llvm-18/bin/clang++",
        "--clang-tidy",
        "--clang-tidy-checks=performance-*,bugprone-*",
        "--all-scopes-completion",
        "--completion-style=detailed",
        "--header-insertion=iwyu",
        "--pch-storage=disk"
    ],
    "clangd.onConfigChanged": "restart",
    "lldb.commandCompletions": true,
    "lldb.dereferencePointers": true,
    "lldb.evaluateForHovers": true,
    "lldb.launch.expressions": "simple",
    "lldb.showDisassembly": "never",
    "lldb.verboseLogging": true,
    // cpp_tools Config
    "C_Cpp.autocomplete": "Disabled",
    "C_Cpp.formatting": "Disabled",
    "C_Cpp.errorSquiggles": "Disabled",
    "C_Cpp.intelliSenseEngine": "Disabled",
    "git.ignoreLimitWarning": true,
}

If the word indexing appears at the bottom, it means the configuration is successful. If it fails, try restarting the window several times.

Debug

Start a stand-alone clickhouse-server instance and perform single-point debugging.
You need to confirm that LLDB is installed.
If the download speed is too slow, you can download it in website and install it manually.

Add launch.json in dir .vscode:

{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Clickhouse Server",
            "type": "lldb",
            "request": "launch",
            "program": "${workspaceFolder}/build/programs/clickhouse",
            "args": [
                "server", "--config-file=${workspaceFolder}/programs/server/config.xml"
            ],
            "initCommands": [
                "process handle -p false -s false -n false SIGUSR1",
                "process handle -p false -s false -n false SIGUSR2"
            ],
            "preLaunchTask": "ninja clickhouse",
            // "stopAtEntry": false,
            // "osx": {
            // "MIMode": "lldb"
            // },
            "cwd": "${workspaceFolder}"
        }
    ]
}

Launch clickhouse-server here, and you can mark some breakpoints to debug.