projgen: Working!

The test-gmod project now builds without any hand-written Makefile, building it by using projgen to generate a makefile then make to build it now works!

I'm pretty happy with it, there's a couple things I might want to fix/add, but for now it's functionally ready.

Now I can probably focus on building a Containerfile for the build image.
This commit is contained in:
Lily Tsuru 2023-07-30 23:44:24 -04:00
parent 568064068e
commit 92a62322f9
8 changed files with 341 additions and 157 deletions

6
.gitignore vendored
View File

@ -4,3 +4,9 @@ module_build/
.cache/
.vscode/
# generated products of the GMOD test program
test-gmod/Makefile
test-gmod/obj
test-gmod/*.elf
test-gmod/*.bin

View File

@ -11,7 +11,7 @@ This is basically the working ideas for the LCPU project.
## Code upload
- Upload a raw binary to execute, generated from any user tooling (goes into server `data/lcpu/users/[steamid]/`)
- Upload a raw binary to execute, generated from any user tooling (goes into the server `data/lcpu/users/[steamid]/`)
- Yes, this means you can run Linux in GMod. No, I'm not sorry.
## Integrated simple project workflow (WIP, not in the addon yet)
@ -23,56 +23,47 @@ This is basically the working ideas for the LCPU project.
- At the root of a project, a `project.json` file is expected to exist, with contents like:
```json
{
"Project": {
"Name": "test",
"Name": "test",
// Define all build configurations here
"Configurations": {
"Debug": {
"CCompileFlags": "-O0 ${BaseCCompileFlags}",
"CppCompileFlags": "-O0 ${BaseCppCompileFlags}"
},
"Release": {
"CCompileFlags": "-O2 ${BaseCCompileFlags}",
"CppCompileFlags": "-O2 ${BaseCppCompileFlags}",
// If a variable is unset it will usually default to
// ${Base${VariableName}}
"LinkerFlags": "-Wl,--gc-sections ${BaseLinkerFlags}"
}
"Configurations": {
"Debug": {
"CCompileFlags": "-O0",
"CppCompileFlags": "-O0",
"LinkerFlags": ""
},
"Release": {
"CCompileFlags": "-O2",
"CppCompileFlags": "-O2",
"LinkerFlags": "-Wl,--gc-sections"
}
},
"Sources": [
"binary.ld",
"start.S",
"main.c"
]
}
"Sources": [
"binary.ld",
"start.S",
"main.c"
]
}
```
- `BaseCCompileFlags` and `BaseCppCompileFlags` are defaulted to sane values for each language.
- This will be transpiled into a `Makefile` by the addon.
- This will be transpiled into a `Makefile` when built.
- A standalone tool will be provided and used for transpiling `project.json` to a `Makefile` (and maybe even built into the container and transpiled there, to reduce the actions on the host to just the podman run?)
- which, when a Build is done in GMod; is then run with `make` in a temporary podman container which only has access to the source code folder for the project (and nothing else, besides riscv tools which are in the image).
- Command line is probably something like `make CONFIG=${config}`
- the output binary will be stored alongside the source code on the server side, with a name like `${name}-${config}.bin`
- which, when a Build is requested in GMod; is then run with `make` in a temporary podman container which only has access to the source code folder for the project (and nothing else, besides riscv tools which are in the image).
- Command line is probably something like `podman run localhost/lcpu-build:latest -v /project:[host project dir] make CONFIG=${config}`
- the output binary will be stored alongside the source code on the server side, with a name like `${name}_${config}.bin`
- This file can then be selected for loading (without needing to be uploaded from the client).
- There is no conditional compilation in the `project.json` system
- All files in a project are always built by that project.
- No notion of subprojects/build dependencies
- This is meant to be simple for easy development in GMod. If you want complex build features you can export the project onto your own computer and use `lcpu_projgen` to generate Makefiles (which you can then maintain)
- Text editor used to edit project source files
- Text editor used to edit project source files in GMod
- Use the Wire editor? (we need wiremod anyways, and the text editor is.. OK I suppose.)
- Or: https://github.com/Metastruct/gmod-monaco
- https://github.com/JustMrPhoenix/Noir/tree/master
- Some example projects?
- A simple bare metal "Hello World"
- A simple bare metal "Hello World" using the SDK headers
- I joke about it, but an RTOS would be really nice and a good stress test of the project system (for usage in "real" projects.)
## Moderation/administration tools

View File

@ -1,8 +1,11 @@
//! These are the hardcoded defaults projgen uses for configuring Makefiles.
//! Only change these if you know what they do.
#pragma once
/// These are the hardcoded defaults
#define PROJGEN_CC "riscv32-unknown-elf-gcc"
#define PROJGEN_CXX "riscv32-unknown-elf-g++"
#define PROJGEN_LD "riscv32-unknown-elf-gcc"
#define PROJGEN_OBJCOPY "riscv32-unknown-elf-objcopy"
#define PROJGEN_BASE_C_FLAGS "-ffreestanding -fno-stack-protector -fdata-sections -ffunction-sections -march=rv32ima -mabi=ilp32"
#define PROJGEN_BASE_CC_FLAGS "-ffreestanding -fno-stack-protector -fdata-sections -ffunction-sections -march=rv32ima -mabi=ilp32"
#define PROJGEN_BASE_CC_FLAGS "-ffreestanding -fno-rtti -fno-exceptions -fno-stack-protector -fdata-sections -ffunction-sections -march=rv32ima -mabi=ilp32"
#define PROJGEN_BASE_LD_FLAGS "-nostdlib"

View File

@ -1,3 +1,7 @@
#pragma once
#include <daw/json/daw_json_link.h>
#include <cstdio>
#include <filesystem>
#include <memory>
@ -7,13 +11,13 @@ namespace fs = std::filesystem;
namespace projgen::util {
using unique_file_ptr = std::unique_ptr<std::FILE, decltype(&std::fclose)>;
using UniqueFilePtr = std::unique_ptr<std::FILE, decltype(&std::fclose)>;
unique_file_ptr UniqueFopen(std::string_view path, std::string_view mode) {
return unique_file_ptr(std::fopen(path.data(), mode.data()), &std::fclose);
inline UniqueFilePtr UniqueFopen(std::string_view path, std::string_view mode) {
return UniqueFilePtr(std::fopen(path.data(), mode.data()), &std::fclose);
}
std::string ReadFileAsString(const fs::path& path) {
inline std::string ReadFileAsString(const fs::path& path) {
auto file = UniqueFopen(path.string(), "r");
std::string data;
if(file) {
@ -27,4 +31,10 @@ namespace projgen::util {
return data;
}
template <class T>
inline T ParseJsonFromFile(const fs::path& path) {
auto data = projgen::util::ReadFileAsString(path);
return daw::json::from_json<T>(data);
}
} // namespace projgen::util

View File

@ -0,0 +1,181 @@
#pragma once
#include <format>
#include <lucore/Assert.hpp>
#include <string>
#include <unordered_map>
#include "BaseConfig.hpp"
#include "FsUtils.hpp"
#include "Project.hpp"
namespace projgen::make {
// base class for makefile rules
struct MakefileGeneratable {
virtual ~MakefileGeneratable() = default;
virtual std::string Generate() = 0;
};
struct MakefileGlobalVariables : public MakefileGeneratable {
std::unordered_map<std::string, std::string> values;
// initalize for the global collection
MakefileGlobalVariables(const std::string& projectName, const std::string& objects) {
values["NAME"] = projectName;
values["CC"] = PROJGEN_CC;
values["CXX"] = PROJGEN_CXX;
values["LD"] = PROJGEN_LD;
values["OBJCOPY"] = PROJGEN_OBJCOPY;
values["BASE_CCFLAGS"] = PROJGEN_BASE_C_FLAGS;
values["BASE_CXXFLAGS"] = PROJGEN_BASE_CC_FLAGS;
values["BASE_LDFLAGS"] = PROJGEN_BASE_LD_FLAGS;
values["OBJS"] = objects;
}
std::string Generate() override {
auto str = std::string();
for(auto& kv : values) {
str += std::format("{} = {}\n", kv.first, kv.second);
}
return str;
}
};
struct MakefileConfiguration : public MakefileGeneratable {
MakefileConfiguration(const std::unordered_map<std::string, projgen::Project::Configuration>& configs) : configs(configs) {}
std::string Generate() override {
auto string = std::string();
for(auto& config : configs) {
string += std::format("ifeq ($(CONFIG),{})\n", config.first);
string += std::format("{}_Valid = yes\n", config.first);
string += std::format("{}_CCFLAGS = $(BASE_CCFLAGS) {}\n", config.first, config.second.cCompileFlags);
string += std::format("{}_CXXFLAGS = $(BASE_CXXFLAGS) {}\n", config.first, config.second.cppCompileFlags);
string += std::format("{}_LDFLAGS = $(BASE_LDFLAGS) {}\n", config.first, config.second.linkerFlags.value_or(""));
string += std::format("endif\n");
}
string += std::format("ifeq ($(CONFIG),)\n$(error Please specify a build configuration)\nendif\n");
string += std::format("ifneq ($($(CONFIG)_Valid), yes)\n$(error Invalid configuration $(CONFIG))\nendif\n");
return string;
}
private:
const std::unordered_map<std::string, projgen::Project::Configuration>& configs;
};
// a general product rule
struct MakefileProductRule : public MakefileGeneratable {
std::string product;
std::string consumeDeps;
MakefileProductRule(const std::string& product, const std::string& consumeDeps) : product(product), consumeDeps(consumeDeps) {}
std::string Generate() override { return std::format("{}: {}\n\t{}", product, consumeDeps, GenerateRuleCommand()); }
private:
virtual std::string GenerateRuleCommand() = 0;
};
struct MakefileAllRule : MakefileGeneratable {
std::string product;
std::string consumeDeps;
MakefileAllRule() : product("all"), consumeDeps("obj/$(CONFIG)/ $(NAME)_$(CONFIG).bin") {}
std::string Generate() override { return std::format("{}: {}\n", product, consumeDeps); }
};
struct MakefileCleanRule : MakefileProductRule {
MakefileCleanRule() : MakefileProductRule("clean", "") {}
private:
std::string GenerateRuleCommand() override { return "rm -rf $(NAME)_$(CONFIG).elf $(NAME)_$(CONFIG).bin obj/$(CONFIG)"; }
};
struct MakefileObjDirRule : MakefileProductRule {
MakefileObjDirRule() : MakefileProductRule("obj/$(CONFIG)/", "") {}
private:
std::string GenerateRuleCommand() override { return "mkdir -p $@"; }
};
// a pattern rule
// e.g "%.o: %.c" or such
struct MakefilePatternRule : public MakefileGeneratable {
std::string productExtension;
std::string consumeExtension;
MakefilePatternRule(const std::string& productExtension, const std::string& consumeExtension)
: productExtension(productExtension), consumeExtension(consumeExtension) {}
std::string Generate() override { return std::format("{}: {}\n\t{}", productExtension, consumeExtension, GenerateRuleCommand()); }
private:
virtual std::string GenerateRuleCommand() = 0;
};
struct MakefileAsmRule : public MakefilePatternRule {
MakefileAsmRule() : MakefilePatternRule("obj/$(CONFIG)/%.o", "%.S") {}
private:
std::string GenerateRuleCommand() override { return "$(CC) -xassembler-with-cpp -c $($(CONFIG)_CCFLAGS) $< -o $@"; }
};
struct MakefileCRule : public MakefilePatternRule {
MakefileCRule() : MakefilePatternRule("obj/$(CONFIG)/%.o", "%.c") {}
private:
std::string GenerateRuleCommand() override { return "$(CC) -c $($(CONFIG)_CCFLAGS) $< -o $@"; }
};
struct MakefileCXXRule : public MakefilePatternRule {
MakefileCXXRule() : MakefilePatternRule("obj/$(CONFIG)/%.o", "%.cpp") {}
private:
std::string GenerateRuleCommand() override { return "$(CXX) -c $($(CONFIG)_CXXFLAGS) $< -o $@"; }
};
struct MakefileLinkRule : public MakefileProductRule {
MakefileLinkRule() : MakefileProductRule("$(NAME)_$(CONFIG).elf", "$(OBJS)") {};
private:
std::string GenerateRuleCommand() override { return "$(LD) $($(CONFIG)_LDFLAGS) $(OBJS) -o $@"; }
};
struct MakefileFlatBinaryRule : MakefileProductRule {
MakefileFlatBinaryRule() : MakefileProductRule("$(NAME)_$(CONFIG).bin", "$(NAME)_$(CONFIG).elf") {};
private:
std::string GenerateRuleCommand() override { return "$(OBJCOPY) $^ -O binary $@"; }
};
struct MakefileWriter {
MakefileWriter(const fs::path& path) {
file = projgen::util::UniqueFopen((path / "Makefile").string(), "w");
LUCORE_CHECK(file, "Could not open {} for writing", (path / "Makefile").string());
}
bool Write(const std::vector<std::unique_ptr<MakefileGeneratable>>& g) {
for(auto& p : g) {
auto generated_data = p->Generate();
if(std::fwrite(generated_data.data(), 1, generated_data.length(), file.get()) != generated_data.length())
return false;
if(!fputc('\n', file.get()))
return false;
}
return true;
}
private:
projgen::util::UniqueFilePtr file { nullptr, std::fclose };
};
} // namespace projgen::make

View File

@ -1,10 +1,12 @@
#include <daw/json/daw_json_link.h>
#pragma once
#include <optional>
#include <string>
#include <unordered_map>
#include <vector>
#include "FsUtils.hpp"
namespace projgen {
struct Project {
struct Configuration {
@ -20,6 +22,52 @@ namespace projgen {
std::vector<std::string> sourceFileNames;
};
// this describes a source file
struct SourceFile {
enum class Type {
Invalid, // invalid file
AsmSourceFile, // Assembly source file
CSourceFile, // C source code
CppSourceFile, // C++ source code
LinkerScript, // prepended to linker flags with a -T (only one can exist in a project)
};
static constexpr Type TypeFromExtension(std::string_view extension) {
if(extension == ".S")
return Type::AsmSourceFile;
else if(extension == ".c")
return Type::CSourceFile;
else if(extension == ".cpp")
return Type::CppSourceFile;
else if(extension == ".ld")
return Type::LinkerScript;
else
return Type::Invalid;
}
explicit SourceFile(const fs::path& path) {
if(path.has_extension())
type = TypeFromExtension(path.extension().string());
else
type = Type::Invalid;
this->path = path.filename();
}
static std::vector<SourceFile> MakeArray(const fs::path& sourcePath, const std::vector<std::string>& filenames) {
auto vec = std::vector<SourceFile>();
vec.reserve(filenames.size());
for(auto& filename : filenames)
vec.emplace_back((sourcePath / filename));
return vec;
}
Type type;
fs::path path;
};
} // namespace projgen
/// DAW JSON Link bindings
@ -36,10 +84,10 @@ namespace daw::json {
template <>
struct json_data_contract<projgen::Project> {
using type =
json_member_list<json_string<"Name">,
json_key_value<"Configurations", std::unordered_map<std::string, projgen::Project::Configuration>, projgen::Project::Configuration>,
json_array<"Sources", std::string> >;
using type = json_member_list<
json_string<"Name">,
json_key_value<"Configurations", std::unordered_map<std::string, projgen::Project::Configuration>, projgen::Project::Configuration>,
json_array<"Sources", std::string> >;
static inline auto to_json_data(const projgen::Project& value) {
return std::forward_as_tuple(value.name, value.configurations, value.sourceFileNames);

View File

@ -1,89 +1,87 @@
//! Main for the LCPU project generator
#include <filesystem>
#include <lucore/Assert.hpp>
#include <lucore/Logger.hpp>
#include <lucore/StdoutSink.hpp>
#include <unordered_map>
#include "BaseConfig.hpp"
#include "FsUtils.hpp"
#include "Makefile.hpp"
#include "Project.hpp"
template <class T>
T ParseJsonFromFile(const fs::path& path) {
auto data = projgen::util::ReadFileAsString(path);
return daw::json::from_json<T>(data);
// TODO:
// Once there's better C++23 ranges support, this can/should be replaced with:
// (objects | views::join_with(' ')
// | std::ranges::to<std::string>())
// I want better ranges support in gcc and clang, this already works in MSVC :(
inline auto join(const std::vector<std::string>& values, const std::string_view seperator = " ") {
auto string = std::string {};
for(auto& value : values)
string += std::format("{}{}", value, seperator);
return string;
}
// this describes a source file
struct SourceFile {
enum class Type {
Invalid, // invalid file
AsmSourceFile, // Assembly source file
CSourceFile, // C source code
CppSourceFile, // C++ source code
LinkerScript, // prepended to linker flags with a -T (only one can exist in a project)
};
static constexpr Type TypeFromExtension(std::string_view extension) {
if(extension == "s" || extension == "S")
return Type::AsmSourceFile;
if(extension == "c")
return Type::CSourceFile;
if(extension == "cpp" || extension == "cc")
return Type::CppSourceFile;
if(extension == "ld")
return Type::LinkerScript;
return Type::Invalid;
}
explicit SourceFile(const std::string& filename) : filename(filename) {
if(auto pos = filename.rfind('.'); pos != std::string::npos) {
type = TypeFromExtension(filename.substr(pos + 1));
} else {
type = Type::Invalid;
}
}
static std::vector<SourceFile> MakeArray(const std::vector<std::string>& filenames) {
auto vec = std::vector<SourceFile>();
vec.reserve(filenames.size());
for(auto& filename : filenames)
vec.emplace_back(filename);
return vec;
}
Type type;
std::string filename;
};
int main(int argc, char** argv) {
lucore::LoggerAttachStdout();
lucore::LogInfo("LCPU project generator!");
lucore::LogInfo("LCPU project generator");
auto project_json_path = (fs::current_path() / "project.json");
if(!fs::exists(project_json_path)) {
lucore::LogFatal("The directory \"{}\" does not seem like it's a project to me", fs::current_path().string());
lucore::LogFatal("The directory \"{}\" does not seem like it's a LCPU project to me", fs::current_path().string());
return 1;
}
try {
auto project = ParseJsonFromFile<projgen::Project>(project_json_path);
auto project = projgen::util::ParseJsonFromFile<projgen::Project>(project_json_path);
for(auto& pair : project.configurations) {
std::printf("%s: %s %s %s\n", pair.first.c_str(), pair.second.cCompileFlags.c_str(), pair.second.cppCompileFlags.c_str(),
pair.second.linkerFlags.value_or(PROJGEN_BASE_LD_FLAGS).c_str());
}
lucore::LogInfo("Generating Makefile for project \"{}\".", project.name);
auto sourceFiles = SourceFile::MakeArray(project.sourceFileNames);
auto sourceFiles = projgen::SourceFile::MakeArray(fs::current_path(), project.sourceFileNames);
auto objects = std::vector<std::string> {};
bool foundLdScript = false;
for(auto& source : sourceFiles) {
std::printf("%s -> %d\n", source.filename.c_str(), source.type);
LUCORE_CHECK(source.type != projgen::SourceFile::Type::Invalid,
"Source file {} with Invalid type in source files. Refusing to generate project", source.path.string());
if(source.type == projgen::SourceFile::Type::LinkerScript) {
LUCORE_CHECK(!foundLdScript, "Project invalid; has more than 1 .ld script file");
foundLdScript = true;
for(auto& kv : project.configurations) {
if(kv.second.linkerFlags.has_value()) {
kv.second.linkerFlags = std::format("-T {} {}", source.path.string(), *kv.second.linkerFlags);
} else {
kv.second.linkerFlags = std::format("-T {}", source.path.string());
}
}
continue;
}
objects.push_back(std::format("obj/$(CONFIG)/{}", source.path.replace_extension(".o").string()));
}
projgen::make::MakefileWriter writer(fs::current_path());
auto generators = std::vector<std::unique_ptr<projgen::make::MakefileGeneratable>> {};
generators.emplace_back(new projgen::make::MakefileGlobalVariables(project.name, join(objects)));
generators.emplace_back(new projgen::make::MakefileConfiguration(project.configurations));
generators.emplace_back(new projgen::make::MakefileAllRule());
generators.emplace_back(new projgen::make::MakefileCleanRule());
generators.emplace_back(new projgen::make::MakefileObjDirRule());
generators.emplace_back(new projgen::make::MakefileAsmRule());
generators.emplace_back(new projgen::make::MakefileCRule());
generators.emplace_back(new projgen::make::MakefileCXXRule());
generators.emplace_back(new projgen::make::MakefileFlatBinaryRule());
generators.emplace_back(new projgen::make::MakefileLinkRule());
if(!writer.Write(generators)) {
lucore::LogError("Could not generate project");
return 1;
} else {
lucore::LogInfo("Generated Makefile \"{}\".", (fs::current_path() / "Makefile").string());
}
} catch(daw::json::json_exception& ex) {

View File

@ -1,53 +0,0 @@
PROJECT = test
# where your rv32 toolchain is
TCPATH = /home/lily/bin/riscv/bin
PREFIX = $(TCPATH)/riscv32-unknown-elf
CC = $(PREFIX)-gcc
CXX = $(PREFIX)-g++
ARCHFLAGS = -ffreestanding -fno-stack-protector -fdata-sections -ffunction-sections -march=rv32ima -mabi=ilp32
CCFLAGS = -g -Os $(ARCHFLAGS) -std=c18
CXXFLAGS = $(ARCHFLAGS) -g -Os -std=c++20 -fno-exceptions -fno-rtti
LDFLAGS = -T binary.ld -nostdlib -Wl,--gc-sections
OBJS = start.o \
main.o
.PHONY: all test clean
all: $(PROJECT).bin $(PROJECT).debug.txt
# this assumes the lcpu project build dir you're using is
# [lcpu repo root]/build
test: $(PROJECT).bin $(PROJECT).debug.txt
../../../../build/projects/riscv_test_harness/rvtest $<
clean:
rm $(PROJECT).elf $(PROJECT).bin $(PROJECT).debug.txt $(OBJS)
# Link rules
$(PROJECT).elf: $(OBJS)
$(CC) $(CCFLAGS) $(LDFLAGS) -o $@ $(OBJS)
$(PROJECT).bin : $(PROJECT).elf
$(PREFIX)-objcopy $^ -O binary $@
$(PROJECT).debug.txt : $(PROJECT).elf
$(PREFIX)-objdump -t $^ > $@
$(PREFIX)-objdump -S $^ >> $@
# Compile rules
%.o: %.cpp
$(CXX) -c $(CXXFLAGS) $< -o $@
%.o: %.c
$(CC) -c $(CCFLAGS) $< -o $@
%.o: %.S
$(CC) -x assembler-with-cpp -march=rv32ima -mabi=ilp32 -c $< -o $@