The wal (Write-Ahead Logging) project aims to provide an efficient log management system.
This system ensures high reliability by using a write-ahead logging mechanism.
- Disk based
- Append only write
The Index struct is encoded into a 20-byte format as follows:
+-------------------+--------------------+-------------------+
| Index (8B) | MetadataOffset (8B)| MetadataSize (4B) |
+-------------------+--------------------+-------------------+
The metadata struct is encoded into format as follows:
// Layout of Data struct:
+----------+-------------+----- ... ----+
| Size (4B) | Index (8B) | LogMetadata |
+----------+-------------+----- ... ----+
// Layout of LogMetadata struct:
+----------------+-------------+----------------+------------+-------------+
| SegmentID (4B) | Size (4B) | Sequence (4B) | CRC (4B) | Offset (8B) |
+----------------+-------------+----------------+------------+-------------+
The metadata struct is encoded into format as follows:
+----- ... ----+
| Payload |
+----- ... ----+
To install the wal package, use the following command:
go get github.com/ISSuh/walTo create a new storage instance, use the NewStorage function:
package main
import (
"fmt"
"github.com/ISSuh/wal"
)
func main() {
options := wal.Options{
Path: "/path/to/log/storage",
SegmentFileSize: 1024 * 1024, // default segment size is 1 GB
SyncAfterWrite: true, // sync file when after wrtie
}
storage, err := wal.NewStorage(options)
if err != nil {
fmt.Printf("failed to create storage: %v", err)
return
}
defer storage.Close()
}To write data to the storage, use the Write method:
data := []byte("example log data")
index, err := storage.Write(data)
if err != nil {
log.Fatalf("failed to write data: %v", err)
}
log.Printf("data written at index: %d", index)To read data from the storage, use the Read method:
readData, err := storage.Read(index)
if err != nil {
log.Fatalf("failed to read data: %v", err)
}
log.Printf("read data: %s", string(readData))To ensure all data is flushed to disk, use the Sync method:
if err := storage.Sync(); err != nil {
log.Fatalf("failed to sync data: %v", err)
}If SyncAfterWrite option is true, don't need to call Sync method
To properly close the storage and release resources, use the Close method:
if err := storage.Close(); err != nil {
log.Fatalf("failed to close storage: %v", err)
}package main
import (
"github.com/ISSuh/wal"
)
func main() {
options := wal.Options{
Path: "/path/to/log",
}
log, err := wal.NewLog(options)
if err != nil {
panic(err)
}
defer log.Close()
// Append a log entry
entry := []byte("log entry")
err = log.Append(entry)
if err != nil {
panic(err)
}
}BenchmarkWrite-11 10000 2501022 ns/op 144 B/op 5 allocs/op
BenchmarkWriteWithSyncAfterWrite-11 10000 7084083 ns/op 144 B/op 5 allocs/op
BenchmarkRead-11 10000 2290 ns/op 160 B/op 6 allocs/opThis project is licensed under the MIT License. See the LICENSE file for details.