CI/CD Pipelines (#2)
continuous-integration/drone/push Build is failing Details

Reviewed-on: #2

Added CI/CD Pipelines:
- Testing during push
- Build and deploy binaries during promote

Added .deb build during deployment

Updated README for core, ui and root workspace
This commit is contained in:
Joe Bellus 2022-10-17 02:59:09 +00:00
parent 5957726421
commit 3c3d43034f
12 changed files with 366 additions and 75 deletions

4
.cargo/config Normal file
View File

@ -0,0 +1,4 @@
[target.x86_64-pc-windows-msvc]
rustflags = ["-Ctarget-feature=+crt-static"]
[target.i686-pc-windows-msvc]
rustflags = ["-Ctarget-feature=+crt-static"]

59
.drone.yml Normal file
View File

@ -0,0 +1,59 @@
kind: pipeline
name: default
steps:
- name: test
image: rust:latest
commands:
- apt-get update
- apt-get -y install libgtk-3-dev libxcb-shape0-dev libxcb-xfixes0-dev
- rustup component add clippy
- cargo clippy
- cargo test --all
- name: deploy
image: rust:latest
commands:
- apt-get update
- apt-get -y install libgtk-3-dev libxcb-shape0-dev libxcb-xfixes0-dev
- cargo build --release
- tar cvzf abacus-linux-amd64.tar.gz -C target/release abacus
- cargo install cargo-deb
- cargo deb -p abacus-ui -o target/release/abacus-amd64.deb
- wget https://dl.min.io/client/mc/release/linux-amd64/mc
- chmod +x mc
- ./mc alias set fivesigma https://objects.5sigma.io $MINIOID $MINIOSECRET
- ./mc cp abacus-linux-amd64.tar.gz fivesigma/public/abacus/abacus-linux-amd64.tar.gz
- ./mc cp target/release/abacus-amd64.deb fivesigma/public/abacus/abacus-amd64.deb
when:
event:
- promote
target:
- staging
- production
---
kind: pipeline
name: windows-build
type: exec
platform:
os: windows
arch: amd64
steps:
- name: test
commands:
- rustup default stable
- cargo test
- name: deploy
commands:
- rustup default stable
- mc alias set fivesigma https://objects.5sigma.io $MINIOID $MINIOSECRET
- cargo build --release --target=x86_64-pc-windows-msvc
- mc cp target\x86_64-pc-windows-msvc\release\abacus.exe fivesigma/public/abacus/abacus.exe
when:
event:
- promote
target:
- staging
- production

75
README.org Normal file
View File

@ -0,0 +1,75 @@
#+OPTIONS: toc:nil
* Abacus
A scratch pad for calcualtions.
Abacus is a lightweight scratch pad for making quick calculations. It is programatic by nature and employs a simple custom scripting language designed around calculating. The scripting lanugage is fully features and supports functions, closures, variable assignment, dataframes, sereis, primitive data types, arrays, maps, etc.
Abacus is similar to a very lightweight version of Jupyter, runs as a single binary, and using only a dozen megs of ram.
The following prebuilt binaries are provided:
- [[https://objects.5sigma.io/public/abacus/abacus-linux-amd64.tar.gz][amd64 Linux Binary]] (tar.gz)
- [[https://objects.5sigma.io/public/abacus/abacus.exe][Windows Executable]] (exe)
- [[https://objects.5sigma.io/public/abacus/abacus-amd64.deb][Linux Debian Package]] (.deb)
[[abacus_demo.png]]
For information on the UI application visit [[abacus-ui/][Abacus UI]]
For a scripting reference see [[./abacus-core#headline-2][Abacus Core]]
* Building the project from source
This project is broken into two projects:
- [[abacus-core][Abacus Core]] - Which contains the scirpt processing and calculation engine.
- [[abacus-ui][Abacus UI]] - The desktop application
** Rust
Download and install the rust toolkit: [[https://rustup.rs]].
** Linux
*** General Dependencies
This is a GTK3 application and as such the GTK3 development library needs to be installed:
#+begin_src sh
sudo apt-get install libgtk-3-dev
#+end_src
You may also need lib-shape and lib-xfixes:
#+begin_src
apt-get -y install libxcb-shape0-dev libxcb-xfixes0-dev
#+end_src
** Launching
Run and build via cargo
#+begin_src sh
cargo run
#+end_src
* Project Status
Abacus is currently in early development. It employs a custom built editor which can be slightly odd at times. A primary focus currently is improving the editing experience and adding additional scripting functionality.
** Road map
- +Editor basics+
- +Scripting basics - Scripting functionality provided by extending the Rhai scripting language+
- +Dataframe support - Pandas style dataframe support via the Polars framework+
- +CSV Support - Importing and processing large CSV files+
- Editor improvements - Improve editor to mirror basic VIM functionality and fix text/cursor related bugs
- Math functions - Implement common mathmatical functions and constants into the scripting language
- Dataframe performance - Reduce the performance cost of converting from the scripting engine to polars
- Web/JSON support - Add HTTP requesting functionality and JSON parsing/processing to the scripting engine
* Shoulders of giants
Dependencies and technology credits:
- [[https://github.com/linebender/druid][Druid UI kit]] - The UI for Abacus is built with the Druid UI kit for rust
- [[https://github.com/pola-rs/polars][Polars]] - Dataframes are processed through the Polars data frame library
- [[https://github.com/rhaiscript/rhai][Rhai]] - The scirpting language for Abacus is derrived by extending the Rhai embedded language

122
abacus-core/README.org Normal file
View File

@ -0,0 +1,122 @@
* Abacus Core
Abacus core provides the scripting and calculation engine for [[../][Abacus]].
For General information visit [[../][Abacus]]
For information on the UI application visit [[../abacus-ui/][Abacus UI]]
* Scripting Reference
** Datatypes
The following data types are supported:
| Name | Note |
| Integer | Any integer constant without a decimal place |
| Float | Any number constant written with a decimal place |
| Array | A list of values of the same type surrounded by [] |
| Map | A dictionary of key value pairs |
| Boolean | True and false constants |
| Series | An array used for fast calculations |
| Dataframe | A map of series used for fast calculations |
| String | An array of characters, uses double quotes |
| Char | A single character, uses single quotes |
** Everything is an expression
Everything can be evaulated as an experession. This means variables can be set to if blocks, loops, or anything else;
#+begin_src rust
let x = if y == "test" { 1 } else { 2 };
#+end_src
** Returning output
Output is rendered based on the returned value of a block. The final value of the block is automatically returned. The return keyword is only needed if an early return is preferable.
#+begin_src rust
if x == 0 {
return 0;
}
if x > 10 {
// note the lack of a semicolon here
x / 5
} else {
// note the lack of a semicolon here
x / 10
}
#+end_src
** Variables
Variables can be assigned using the let keyword. Variables are dynamically typed and globally mutable:
#+begin_src rust
let pulse_rate = 0.8;
let time = 3.2;
time / pulse_rate
#+end_src
** Functions
Functions can be defined using the fn keyword. Return types and arguments are dynamic and do not require typing.
#+begin_src rust
fn add_one(x) {
x + 1;
}
add_one(2)
#+end_src
** Object maps
Dictionary style data maps are possible and can be index or accessed by a property notation. Dictionaries are defined using a special syntax wrapper: #{}.
#+begin_src rust
let account = #{ first_name: "Alice", last_name: "Allison", balance: 150.32 };
// Access via property notation
account.balance += 10;
// Access via indexing
account["last_name"] = "Test";
account
#+end_src
** Series
A series provides a powerful way to perform calculations on an array of data. Operations performedon the series are performed against all of its member values. Serieses are constructed using the series fucntion and passing an array of values.
#+begin_src rust
let s = series([1,2,3,4]);
// Return an array of values calculated from the series
[ s + 10, s * s ]
#+end_src
** Dataframes
Dataframes provide an object for working with tabular data made up of several series. Dataframes are initialzied using the dataframe constructor function and passing in an object map container name/array pairs.
Dataframe series can be accessed via indexing or property notation just like object maps.
#+begin_src rust
let df = dataframe(#{
name: ["Alice", "Bob", "Charles"],
rate: [18,20,20],
hours: [22,40,55]
});
df.balance = df.rate * df.hours;
df
#+end_src
*** Filtering
Dataframes are more powerful than object maps and can be filtered. Filtering requires two parts:
1. Selecting the columns to be included
2. providing a predicate to filter a column's value
The filtering syntax is FROM <dataframe> <column array> : <array of predicates>
Predicates consist of a quoted name for the series/column an operator and a value. Valid operators are: gt, gte, lt, lte.
Filtering returns a new dataframe with the extracted values.
#+begin_src rust
let df = dataframe(#{
name: ["Alice", "Bob", "Charles"],
rate: [18,20,20],
hours: [22,40,55]
});
let high_rates = from df ["name", "rate", "hours"] : ["rate" gte 20];
high_rates.balance = high_rates.rate * high_rates.hours;
high_rates
#+end_src

View File

@ -37,6 +37,8 @@ pub fn setup_engine(engine: &mut rhai::Engine) {
engine.register_fn("-", script_functions::subtract_series_i64);
engine.register_fn("-", script_functions::subtract_series_f64);
engine.register_fn("**", script_functions::power_series_f64);
engine.register_fn("**", script_functions::power_series_i64);
engine.register_fn("*", script_functions::multiply_series_series);
engine.register_fn("*", script_functions::multiply_series_i64);
engine.register_fn("*", script_functions::multiply_series_f64);
@ -283,6 +285,34 @@ mod script_functions {
Series(a.0 - b.0)
}
pub fn power_series_i64(a: Series, b: i64) -> ScriptResult<Series> {
let name = a.name();
let a_series = a.0.clone();
let df = a_series
.into_frame()
.lazy()
.select([polars::prelude::col(name).pow(b)])
.collect()
.map_err(|e| e.to_string())?;
let s = df.column(name).map_err(|e| e.to_string())?;
Ok(Series(s.clone()))
}
pub fn power_series_f64(a: Series, b: f64) -> ScriptResult<Series> {
let name = a.name();
let a_series = a.0.clone();
let df = a_series
.into_frame()
.lazy()
.select([polars::prelude::col(name).pow(b)])
.collect()
.map_err(|e| e.to_string())?;
let s = df.column(name).map_err(|e| e.to_string())?;
Ok(Series(s.clone()))
}
pub fn multiply_series_i64(a: Series, b: i64) -> Series {
Series(a.0 * b)
}

View File

@ -3,6 +3,10 @@ name = "abacus-ui"
version = "0.1.0"
edition = "2021"
[[bin]]
name = "abacus"
path = "src/main.rs"
# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
[dependencies]
@ -10,4 +14,20 @@ druid = { git = "https://github.com/linebender/druid.git", features=["im", "svg"
abacus-core = { path = "../abacus-core" }
syntect = "5.0.0"
ropey = "1.5.0"
clipboard = "0.5.0"
clipboard = "0.5.0"
[package.metadata.deb]
name = "Abacus"
maintainer = "Joe Bellus <joe@5sigma.io>"
copyright = "2022, Joe Bellus"
extended-description = """\
A programable scratchpad for quick calcualtions\
"""
depends = "$auto"
section = "utility"
priority = "optional"
assets = [
["target/release/abacus", "usr/bin/", "755"],
["README.org", "usr/share/doc/abacus/README", "644"],
]

47
abacus-ui/README.org Normal file
View File

@ -0,0 +1,47 @@
* Abacus UI
Abacus UI is the desktop GUI for [[../][Abacus]].
For General information visit [[../][Abacus]]
For information on the scripting visit [[../abacus-core/][Abacus Core]]
* Blocks
The editor is separated into multiple /BLOCKS/. These blocks can be used to perform separate, related calculations. Each block can have a single output, which is the returned value from the block.
All blocks share the same global scope, meaning variables, functions, etc defined in a block are available to subsequent blocks.
* Modal editing
Abacus's editor is a modal editor that tries to follow VIM keybinds. The editor has two functional modes:
** Normal mode
In normal mode the cursor is a block and functional keybinds can be used for movement.
*** Normal mode keybinds
| Key | Action |
| i | Enter insert mode |
| h | Cursor left |
| j | Cursor down |
| k | Cursor up |
| l | Cursor right |
| A | Cursor to end of line and enter insert mode |
| I | Cursor to beginning of line and enter insert mode |
| O | Insert new line below and enter insert mode |
| o | Insert new line above and enter insert mode |
| b | Scan backward a word |
| w | Scan forward a word |
| v | Mark selection |
| x | Delete current character or selection |
** Insert mode
In insert mode the cursor is a line and text can be edited. To return to normal mode use the ESC key.
* General Keybinds
These keybinds can be used in any mode
| Key | Action |
| CTRL+C | Copy current selection to clipboard |
| CTRL+V | Paste clipboard at cursor |
| CTRL+N | Create a new block |
| CTRL+ENTER | Run all blocks |
| SHIFT+ENTER | Run the current block |

View File

@ -330,48 +330,5 @@ impl EditorData {
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn cursor_left_normal_empty_line() {
let mut data = EditorData {
mode: EditMode::Normal,
..Default::default()
};
data.push_str("0123456789\n\n1234");
data.cursor_pos = 12;
data.cursor_left();
assert_eq!(data.cursor_pos, 11);
data.cursor_left();
assert_eq!(data.cursor_pos, 10);
}
#[test]
fn cursor_left_normal_double_empty_line() {
let mut data = EditorData {
mode: EditMode::Normal,
..Default::default()
};
data.push_str("0123456789\n\n\n1234");
data.cursor_pos = 14;
data.cursor_left();
assert_eq!(data.cursor_pos, 12);
data.cursor_left();
assert_eq!(data.cursor_pos, 11);
data.cursor_left();
assert_eq!(data.cursor_pos, 10);
}
#[test]
fn cursor_left_normal_end_of_line() {
let mut data = EditorData {
mode: EditMode::Normal,
..Default::default()
};
data.push_str("0123456789\n1234");
data.cursor_pos = 12;
assert_eq!(data.current_char().unwrap(), '1');
data.cursor_left();
assert_eq!(data.cursor_pos, 10);
}
// use super::*;
}

View File

@ -1,7 +1,8 @@
use abacus_core::Output;
use clipboard::ClipboardProvider;
use druid::{
piet::{CairoTextLayout, Text, TextAttribute, TextLayout, TextLayoutBuilder},
piet::{PietTextLayout, Text, TextAttribute, TextLayout, TextLayoutBuilder},
widget::{Container, Flex, Label, Padding, Svg, SvgData},
Color, Event, FontDescriptor, FontFamily, FontWeight, LifeCycle, PaintCtx, Rect, RenderContext,
Target, Widget, WidgetExt,
@ -114,7 +115,7 @@ impl Default for AbacusEditor {
}
impl AbacusEditor {
fn paint_cursor(&self, ctx: &mut PaintCtx, data: &EditorData, layout: &CairoTextLayout) {
fn paint_cursor(&self, ctx: &mut PaintCtx, data: &EditorData, layout: &PietTextLayout) {
if data.mode == EditMode::Insert {
if data.cursor_pos == 0 {
let rects = layout.rects_for_range(0..1);
@ -189,7 +190,7 @@ impl AbacusEditor {
&self,
ctx: &mut PaintCtx,
data: &EditorData,
) -> CairoTextLayout {
) -> PietTextLayout {
let syntax = self.syntax_set.find_syntax_by_extension("rs").unwrap();
let mut h = HighlightLines::new(syntax, &self.theme_set.themes["base16-mocha.dark"]);

View File

@ -1,3 +1,5 @@
#![windows_subsystem = "windows"]
mod app_delegate;
mod app_header;
mod commands;
@ -39,6 +41,7 @@ fn main() -> Result<(), PlatformError> {
AppLauncher::with_window(
WindowDesc::new(build_ui())
.resizable(true)
.title("Abacus")
.window_size((600.0, 800.0)),
)
.delegate(app_delegate::Delegate)

BIN
abacus_demo.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 43 KiB

View File

@ -1,27 +0,0 @@
kind: pipeline
name: default
steps:
- name: test
image: rust:latest
commands:
- rustup component add clippy
- cargo clippy
- cargo test --all
- name: deploy
image: rust:latest
commands:
- cargo build --release
- tar cvzf conductor.tar.gz -C target/release conductor
- wget https://dl.min.io/client/mc/release/linux-amd64/mc
- chmod +x mc
- ./mc alias set fivesigma https://objects.5sigma.io $MINIOID $MINIOSECRET
- ./mc cp conductor.tar.gz fivesigma/public/conductor.tar.gz
# when:
# event:
# - promote
# target:
# - staging
# - production