The following post gives an overview of the open-source verification tools used as part of the Summer of FPGAs -- OrangeCrab Dev Bd - Review  I did for element14 (link to road-test).

Tools

CoCoTB (verification)

CoCoTB is a set of Python modules that in conjunction with an HDL (Hardware Description Language) verification tool can be used to verify and test hardware design using Python.

cocotb is a COroutine based COsimulation TestBench environment for verifying VHDL and SystemVerilog RTL using Python.

The main characteristics to choose CoCoTB over other options are:

  • Python

    import gravity with Python

    • you can code your tests in Python which I found is faster and easier than UVM (Universal Verification Methodology), SystemVerilog (SV),  VHDL or similar verification languages (e, SystemC, ...)
    • you can leverage thousands of available Python libraries, code, and examples - system models, you name it.
  • Simulator independent
    • It is not tied to a particular simulator, and as it is open source, so you can add another simulator if you ever have to.

CoCoTB has several optional modules. Among them, I use the cocotb-test. cocotb-test adds pytest capabilities and removes the need for Makefile(s). Enabling pytest added an extra feature of running simulations (tests) in parallel (pytest-xdist or pytest-parallel).

As a final note, I want to clarify that everything has its place. I think SV or VHDL are OK for some basic unit testing, and UVM is a complex set of SV modules that should be used to test complex systems. But for most of the part, a CoCoTB test suite should suffice. Unless there is a particular need, I personally would stay with CoCoTB.

 

Verilator (simulator)

Verilator is the fastest Verilog/SystemVerilog simulator, and is open-source and free. It is widely used and has a growing community.

Verilator converts a SV design into a C++ (or SystemC) model which can be wrapped with a C/C++ testbench. But in this case, the wrapper is done in Python with CoCoTB handling the TB <-> RTL interface.

GTKwave (waveform viewer)

GTKwave is the de facto HDL simulation waveform viewer in the open-source community. For good or for bad, I do not know any decent alternatives. If you can recommend a better one, just left a comment.

Installation

CoCoTB installation

sudo apt-get install make gcc g++ python3 python3-dev python3-pip
pip install cocotb
pip install cocotb-test
Verilator installation

I install the tools usually under ~/tools/ directory

cd ~
mkdir tools
cd tools

 

Prerequisites

# Prerequisites:
sudo apt-get install git perl python3 make autoconf g++ flex bison ccache
sudo apt-get install libgoogle-perftools-dev numactl perl-doc

sudo apt-get install libfl2     # Ubuntu only (ignore if gives error)
sudo apt-get install libfl-dev  # Ubuntu only (ignore if gives error)

sudo apt-get install zlibc zlib1g zlib1g-dev  # Ubuntu only (ignore if gives error)

git clone https://github.com/verilator/verilator

Patching Verilator for CoCoTB

To use the latest Verilator version in CoCoTB, the following patch is required: VPI: simulation time callbacks should be one-time #2778 at the moment

# vim instructions (any other editor should suffice)
vim verilator/include/verilated_vpi.cpp
:603  # Goto line 603

             VerilatedVpiCbHolder& ho = *it;
             VL_DEBUG_IF_PLI(VL_DBG_MSGF("- vpi: reason_callback reason=%d id=%" VL_PRI64 "d\n",
                                         reason, ho.id()););
>>>>>
add next line =>
             ho.invalidate();
<<<<<
             (ho.cb_rtnp())(ho.cb_datap());
             called = true;
             if (was_last) break;

From the link above:

diff --git a/include/verilated_vpi.cpp b/include/verilated_vpi.cpp
index c354d89f..64099b98 100644
--- a/include/verilated_vpi.cpp
+++ b/include/verilated_vpi.cpp
@@ -603,6 +603,7 @@ public:
             VerilatedVpiCbHolder& ho = *it;
             VL_DEBUG_IF_PLI(VL_DBG_MSGF("- vpi: reason_callback reason=%d id=%" VL_PRI64 "d\n",
                                         reason, ho.id()););
+            ho.invalidate();
             (ho.cb_rtnp())(ho.cb_datap());
             called = true;
             if (was_last) break;

Building Verilator

unsetenv VERILATOR_ROOT  # For csh; ignore error if on bash
unset VERILATOR_ROOT  # For bash

cd ~/tools/verilator
git pull        # Make sure git repository is up-to-date
git tag         # See what versions exist
#git checkout master      # Use development branch (e.g. recent bug fixes)
#git checkout stable      # Use most recent stable release
#git checkout v{version}  # Switch to specified release version

autoconf        # Create ./configure script
./configure     # Configure and create Makefile
make -j         # Build Verilator itself
sudo make install

GTKwave installation

sudo apt install gtkwave

Simple example

SV code

module dummy_multiplier
#(
    parameter WL = 32,
    parameter DONE_PIPE_LVL = 4
)
(
    input wire clk,
    input wire reset,
    
    // Inputs
    input wire          start,
    input wire [WL-1:0] multiplier,
    input wire [WL-1:0] multiplicand,
    
    // Outputs
    output wire done,
    output wire [2*WL-1:0] product
);
    
    logic [DONE_PIPE_LVL-1:0] done_dummy_pipe;
    logic          [2*WL-1:0] product_dummy_pipe[DONE_PIPE_LVL];

    always_ff @(posedge clk) begin : tx_block
        if(reset) begin
            done_dummy_pipe <= '0;
            
        end else begin
            product_dummy_pipe[0] <= multiplier * multiplicand;
            for(int i=1; i<DONE_PIPE_LVL; ++i)
                product_dummy_pipe[i] <= product_dummy_pipe[i-1];
            done_dummy_pipe <= {done_dummy_pipe[DONE_PIPE_LVL-2:0], start};
        end
    end
    assign done = done_dummy_pipe[DONE_PIPE_LVL-1];
    assign product = product_dummy_pipe[DONE_PIPE_LVL-1];
endmodule

CoCoTB wrapper

import os
import logging
from pathlib import Path
import cocotb
from cocotb.clock import Clock
from cocotb.triggers import FallingEdge, RisingEdge, Timer
from cocotb_test import simulator
# -----------------------------------------------------------------------------
# CoCoTB Module
@cocotb.test()
async def dummy_multiplier_basic_test(dut):
    """
    Basic multiplier test
    """
    # Set logger
    log = logging.getLogger("TB")
    log.setLevel('INFO')
    # Start test
    log.info('Multiplier basic test')
    # Setup TB
    clock = Clock(dut.clk, 10, units='ns') 
    cocotb.fork(clock.start())
    # Reset system
    await FallingEdge(dut.clk)
    log.info('Set reset')
    dut.reset <= 1
    await RisingEdge(dut.clk)
    log.info('Release reset')
    dut.reset <= 0
    # Set inputs
    log.info("Doing 5x7 = 35")
    dut.multiplier   <= 5
    dut.multiplicand <= 7
    dut.start <= 1
    await RisingEdge(dut.clk)
    dut.start <= 0
    # Wait result
    max_cnt = 10
    while dut.done == 0 and max_cnt:
        log.info("Waiting DUT ready")
        max_cnt -= 1
        await RisingEdge(dut.clk)
    
    if(max_cnt != 0):
        log.info(f'Data captured: {dut.product}')
        assert dut.product == 35, f'5x7 -> 35, but got {dut.product.value}'
    else:
        log.error(f'max_cnt reached without done set')
        assert False
    
    # Sim done
    await Timer(100, units='ns')
    log.info('Test done!')
# -----------------------------------------------------------------------------
# Invoke test
if __name__ == '__main__':
    # cocotb-test default is icarus, switching to verilator
    sim = os.environ.get('SIM', 'verilator')
    os.environ['SIM'] = sim
    # ---------------------------------------
    _workpath = str(Path(__file__).resolve().parent)
    # ---------------------------------------
    sim_build = str(Path(_workpath) / "sim_build" / 'dummy_multiplier_basic_test')
    # ---------------------------------------
    simulator.run(
        toplevel='dummy_multiplier',
        module=Path(__file__).stem,
        verilog_sources=['../rtl/dummy_multiplier.sv'],
        sim_build=sim_build,
    )

 

Running the test

WAVES=1 python multiplier_simple_tb.py

Console output

INFO cocotb: Running command: perl /usr/local/bin/verilator -cc --exe -Mdir /home/dramoz/dev/sv_blocks/tb/sim_build/dummy_multiplier_basic_test -DCOCOTB_SIM=1 --top-module dummy_multiplier --vpi --public-flat-rw --prefix Vtop -o dummy_multiplier -LDFLAGS -Wl,-rpath,/home/dramoz/.virtualenvs/cocotb/lib/python3.8/site-packages/cocotb/libs -L/home/dramoz/.virtualenvs/cocotb/lib/python3.8/site-packages/cocotb/libs -lcocotbvpi_verilator --trace-fst --trace-structs /home/dramoz/.virtualenvs/cocotb/lib/python3.8/site-packages/cocotb/share/lib/verilator/verilator.cpp /home/dramoz/dev/sv_blocks/rtl/dummy_multiplier.sv
INFO cocotb: Running command: make -C /home/dramoz/dev/sv_blocks/tb/sim_build/dummy_multiplier_basic_test -f Vtop.mk
INFO cocotb: make: Entering directory '/home/dramoz/dev/sv_blocks/tb/sim_build/dummy_multiplier_basic_test'
INFO cocotb: ccache g++  -std=c++11 -I.  -MMD -I/usr/local/share/verilator/include -I/usr/local/share/verilator/include/vltstd -DVM_COVERAGE=0 -DVM_SC=0 -DVM_TRACE=1 -DVM_TRACE_FST=1 -faligned-new -fcf-protection=none -Wno-bool-operation -Wno-sign-compare -Wno-uninitialized -Wno-unused-but-set-variable -Wno-unused-parameter -Wno-unused-variable -Wno-shadow      -std=c++14 -Os -c -o verilator.o /home/dramoz/.virtualenvs/cocotb/lib/python3.8/site-packages/cocotb/share/lib/verilator/verilator.cpp
INFO cocotb: ccache g++  -std=c++11 -I.  -MMD -I/usr/local/share/verilator/include -I/usr/local/share/verilator/include/vltstd -DVM_COVERAGE=0 -DVM_SC=0 -DVM_TRACE=1 -DVM_TRACE_FST=1 -faligned-new -fcf-protection=none -Wno-bool-operation -Wno-sign-compare -Wno-uninitialized -Wno-unused-but-set-variable -Wno-unused-parameter -Wno-unused-variable -Wno-shadow      -std=c++14 -Os -c -o verilated.o /usr/local/share/verilator/include/verilated.cpp
INFO cocotb: ccache g++  -std=c++11 -I.  -MMD -I/usr/local/share/verilator/include -I/usr/local/share/verilator/include/vltstd -DVM_COVERAGE=0 -DVM_SC=0 -DVM_TRACE=1 -DVM_TRACE_FST=1 -faligned-new -fcf-protection=none -Wno-bool-operation -Wno-sign-compare -Wno-uninitialized -Wno-unused-but-set-variable -Wno-unused-parameter -Wno-unused-variable -Wno-shadow      -std=c++14 -Os -c -o verilated_dpi.o /usr/local/share/verilator/include/verilated_dpi.cpp
INFO cocotb: ccache g++  -std=c++11 -I.  -MMD -I/usr/local/share/verilator/include -I/usr/local/share/verilator/include/vltstd -DVM_COVERAGE=0 -DVM_SC=0 -DVM_TRACE=1 -DVM_TRACE_FST=1 -faligned-new -fcf-protection=none -Wno-bool-operation -Wno-sign-compare -Wno-uninitialized -Wno-unused-but-set-variable -Wno-unused-parameter -Wno-unused-variable -Wno-shadow      -std=c++14 -Os -c -o verilated_vpi.o /usr/local/share/verilator/include/verilated_vpi.cpp
INFO cocotb: ccache g++  -std=c++11 -I.  -MMD -I/usr/local/share/verilator/include -I/usr/local/share/verilator/include/vltstd -DVM_COVERAGE=0 -DVM_SC=0 -DVM_TRACE=1 -DVM_TRACE_FST=1 -faligned-new -fcf-protection=none -Wno-bool-operation -Wno-sign-compare -Wno-uninitialized -Wno-unused-but-set-variable -Wno-unused-parameter -Wno-unused-variable -Wno-shadow      -std=c++14 -Os -c -o verilated_fst_c.o /usr/local/share/verilator/include/verilated_fst_c.cpp
INFO cocotb: /usr/bin/perl /usr/local/share/verilator/bin/verilator_includer -DVL_INCLUDE_OPT=include Vtop.cpp Vtop___024root__DepSet_heccd7ead__0.cpp Vtop__Dpi.cpp Vtop__Trace__0.cpp Vtop___024root__Slow.cpp Vtop___024root__DepSet_heccd7ead__0__Slow.cpp Vtop__Syms.cpp Vtop__Trace__0__Slow.cpp > Vtop__ALL.cpp
INFO cocotb: ccache g++  -std=c++11 -I.  -MMD -I/usr/local/share/verilator/include -I/usr/local/share/verilator/include/vltstd -DVM_COVERAGE=0 -DVM_SC=0 -DVM_TRACE=1 -DVM_TRACE_FST=1 -faligned-new -fcf-protection=none -Wno-bool-operation -Wno-sign-compare -Wno-uninitialized -Wno-unused-but-set-variable -Wno-unused-parameter -Wno-unused-variable -Wno-shadow      -std=c++14 -Os -c -o Vtop__ALL.o Vtop__ALL.cpp
INFO cocotb: echo "" > Vtop__ALL.verilator_deplist.tmp
INFO cocotb: Archive ar -rcs Vtop__ALL.a Vtop__ALL.o
INFO cocotb: g++    verilator.o verilated.o verilated_dpi.o verilated_vpi.o verilated_fst_c.o Vtop__ALL.a   -Wl,-rpath,/home/dramoz/.virtualenvs/cocotb/lib/python3.8/site-packages/cocotb/libs -L/home/dramoz/.virtualenvs/cocotb/lib/python3.8/site-packages/cocotb/libs -lcocotbvpi_verilator -lz    -o dummy_multiplier
INFO cocotb: rm Vtop__ALL.verilator_deplist.tmp
INFO cocotb: make: Leaving directory '/home/dramoz/dev/sv_blocks/tb/sim_build/dummy_multiplier_basic_test'
INFO cocotb: Running command: /home/dramoz/dev/sv_blocks/tb/sim_build/dummy_multiplier_basic_test/dummy_multiplier
INFO cocotb:      -.--ns INFO     cocotb.gpi                         ..mbed/gpi_embed.cpp:100  in set_program_name_in_venv        Using Python virtual environment interpreter at /home/dramoz/.virtualenvs/cocotb/bin/python
INFO cocotb:      -.--ns INFO     cocotb.gpi                         ../gpi/GpiCommon.cpp:105  in gpi_print_registered_impl       VPI registered
INFO cocotb:      -.--ns INFO     cocotb.gpi                         ..mbed/gpi_embed.cpp:240  in _embed_sim_init                 Python interpreter initialized and cocotb loaded!
INFO cocotb:      0.00ns INFO     cocotb                                      __init__.py:220  in _initialise_testbench_          Running on Verilator version 4.212 2021-09-01
INFO cocotb:      0.00ns INFO     cocotb                                      __init__.py:226  in _initialise_testbench_          Running tests with cocotb v1.5.2 from /home/dramoz/.virtualenvs/cocotb/lib/python3.8/site-packages/cocotb
INFO cocotb:      0.00ns INFO     cocotb                                      __init__.py:247  in _initialise_testbench_          Seeding Python random module with 1632919199
INFO cocotb:      0.00ns INFO     cocotb.regression                         regression.py:127  in __init__                        Found test multiplier_simple_tb.dummy_multiplier_basic_test
INFO cocotb:      0.00ns INFO     cocotb.regression                         regression.py:468  in _start_test                     Running test 1/1: dummy_multiplier_basic_test
INFO cocotb:      0.00ns INFO     ..tiplier_basic_test.0x7fe425b2b310       decorators.py:312  in _advance                        Starting test: "dummy_multiplier_basic_test"
INFO cocotb:                                                                                                                      Description:
INFO cocotb:                                                                                                                          Basic multiplier test
INFO cocotb: 
INFO cocotb:      0.00ns INFO     TB                                 ..plier_simple_tb.py:39   in dummy_multiplier_basic_test     Multiplier basic test
INFO cocotb:      5.00ns INFO     TB                                 ..plier_simple_tb.py:47   in dummy_multiplier_basic_test     Set reset
INFO cocotb:     10.00ns INFO     TB                                 ..plier_simple_tb.py:50   in dummy_multiplier_basic_test     Release reset
INFO cocotb:     10.00ns INFO     TB                                 ..plier_simple_tb.py:54   in dummy_multiplier_basic_test     Doing 5x7 = 35
INFO cocotb:     20.00ns INFO     TB                                 ..plier_simple_tb.py:64   in dummy_multiplier_basic_test     Waiting DUT ready
INFO cocotb:     30.00ns INFO     TB                                 ..plier_simple_tb.py:64   in dummy_multiplier_basic_test     Waiting DUT ready
INFO cocotb:     40.00ns INFO     TB                                 ..plier_simple_tb.py:64   in dummy_multiplier_basic_test     Waiting DUT ready
INFO cocotb:     50.00ns INFO     TB                                 ..plier_simple_tb.py:64   in dummy_multiplier_basic_test     Waiting DUT ready
INFO cocotb: /home/dramoz/dev/sv_blocks/tb/multiplier_simple_tb.py:69: FutureWarning: `str(ModifiableObject)` is deprecated, and in future will return `ModifiableObject._path`. To get a string representation of the value, use `str(ModifiableObject.value)`.
INFO cocotb:   log.info(f'Data captured: {dut.product}')
INFO cocotb:     60.00ns INFO     TB                                 ..plier_simple_tb.py:69   in dummy_multiplier_basic_test     Data captured: 0000000000000000000000000000000000000000000000000000000000100011
INFO cocotb:    160.00ns INFO     TB                                 ..plier_simple_tb.py:77   in dummy_multiplier_basic_test     Test done!
INFO cocotb:    160.00ns INFO     cocotb.regression                         regression.py:364  in _score_test                     Test Passed: dummy_multiplier_basic_test
INFO cocotb:    160.00ns INFO     cocotb.regression                         regression.py:487  in _log_test_summary               Passed 1 tests (0 skipped)
INFO cocotb:    160.00ns INFO     cocotb.regression                         regression.py:557  in _log_test_summary               **********************************************************************************************************
INFO cocotb:                                                                                                                      ** TEST                                              PASS/FAIL  SIM TIME(NS)  REAL TIME(S)  RATIO(NS/S) **
INFO cocotb:                                                                                                                      **********************************************************************************************************
INFO cocotb:                                                                                                                      ** multiplier_simple_tb.dummy_multiplier_basic_test    PASS          160.00          0.01     11913.81  **
INFO cocotb:                                                                                                                      **********************************************************************************************************
INFO cocotb: 
INFO cocotb:    160.00ns INFO     cocotb.regression                         regression.py:574  in _log_sim_summary                *************************************************************************************
INFO cocotb:                                                                                                                      **                                 ERRORS : 0                                      **
INFO cocotb:                                                                                                                      *************************************************************************************
INFO cocotb:                                                                                                                      **                               SIM TIME : 160.00 NS                              **
INFO cocotb:                                                                                                                      **                              REAL TIME : 0.05 S                                 **
INFO cocotb:                                                                                                                      **                        SIM / REAL TIME : 3549.59 NS/S                           **
INFO cocotb:                                                                                                                      *************************************************************************************
INFO cocotb: 
INFO cocotb:    160.00ns INFO     cocotb.regression                         regression.py:259  in tear_down                       Shutting down...
INFO cocotb: - :0: Verilog $finish
INFO cocotb: - /home/dramoz/.virtualenvs/cocotb/lib/python3.8/site-packages/cocotb/share/lib/verilator/verilator.cpp:118: Verilog $finish
INFO cocotb: - /home/dramoz/.virtualenvs/cocotb/lib/python3.8/site-packages/cocotb/share/lib/verilator/verilator.cpp:118: Second verilog $finish, exiting
Results file: /home/dramoz/dev/sv_blocks/tb/sim_build/dummy_multiplier_basic_test/_r77l7fe_results.xml

 

View wave

gtkwave sim_build/dummy_multiplier_basic_test/dump.fst

 

dummy multiplier tb wave