Programming

Creating a Simple Distibuted Search Engine Using Worker Pool on Golang

  • avatar
    by Ihsan
  • 23 Apr 2021
  • 3 minutes

Introduction

You know Google is a place where we search for several keywords that can produce various millions of information that we can get just by typing a few keywords and in less than 1 second we can already get various information related to the keywords we are looking for. This is an example of a miniature or simple search engine version of how we can implement a worker pool into an engine that is useful for searching.

Do you understand how Worker Pool works? If you don’t know or already know but want to deepen it again, you can read first in the previous article that Santekno discussed in how to implement concurrency worker pool golang.

Basic Data Preparation

Before we create a search program, we must first prepare the data to be searched. Santekno here will provide the simplest data sample so that it is easy for everyone to digest.

We will create data in the form of User information which contains name and email attributes. This name and email will be the data that we will search.

type User struct {
	Name string
	Email string
}

var Database = []User{
	{ Name: "ihsan arif", Email: "ihsan.arif@example.com"},
	{Name: "ihsan nugraha", Email: "ihsa@example.com"},
	{Name: "arif susanto", Email: "arif.susanto@example.com"},
	{Name: "ines nur", Email: "ines.nur@example.com"},
	{Name: "inez hendriani", Email: "inez.hendriani@example.com"},
	{Name: "deni siahaan", Email: "deni.siahaan@example.com"},
	{Name: "sumarno treggono", Email: "sumarno.trenggono@example.com"},
	{Name: "nida fauziah", Email: "nida.fauziah@example.com"},
	{Name: "nurul kamilah", Email: "nurul.kamilah@example.com"},
	{Name: "muhammad rafi", Email: "muhammad.rafi@example.com"},
	{Name: "irfan maulana", Email: "irfan.maulana@example.com"},
	{Name: "irsyad aprilianto", Email: "irsyad.aprilianto@example.com"},
	{Name: "iqbal fauzi", Email: "iqbal.fauzi@example.com"},
	{Name: "rani zahroh", Email: "rani.zahroh@example.com"},
	{Name: "ajeng syifa", Email: "ajeng.syifa@example.com"},
	{Name: "aldi arikandi", Email: "aldi.arikandi@example.com"},
	{Name: "firman shah", Email: "firman.syah@example.com"},
	{Name: "irwan setiawan", Email: "irwan.setiawan@example.com"},
}

Worker Pool Initialization

Why do we use worker pool? The simple answer is so that the search process based on keywords into the database (data source) is faster and shared. We can see below the initialization of Worker struct into an object that we will use for searching in the main function. If you are still confused, just look at the program snippet for the Woerker initialization.

type Worker struct {
	Users []User
	ch chan *User
	Name string
}

func NewWorker(users []User, ch chan *User, name string) *Worker {
	return &Worker{Users: users, ch: ch, Name: name}
}

Search Function

We will create the main function of the worker under the worker object to make searching easier.

func (w *Worker) Find(email string) *User {
	for i := range w.Users {
		user := &w.Users[i]
		if strings.Contains(user.Email, email) {
			w.ch <- user
		}
	}
	return nil
}

Main Program Function

This function runs the process of initializing the worker, retrieving data and reading the input to be the keyword to be searched in the database.

func main() {
    // to retrieve input arguments
	email := os.Args[1]
    // initialize the user object channel
	ch := make(chan *User)
    // initialize worker pool with 3 pools 
	for i := 0; i < 3; i++ {
		go NewWorker(Database[(len(Database)/3*i):len(Database)/3*(i+1)], ch, fmt.Sprintf("#%s", i)).Find(email)
	}

    // print the result of the search
	select {
	case user := <-ch:
		log.Printf("the email %s is owned by %s", email, user.Email)
	case < time.After(100 * time.Millisecond):
		log.Printf("the email %s was not found", email)
	}
}

Program Output

-> go run content/post/main.go ihsan.arif
2021/05/10 19:38:00 the email ihsan.arif is owned by ihsan.arif@example.com
-> go run content/post/main.go ines      
2021/05/10 19:38:04 the email ines is owned by ines.nur@example.com
-> go run content/post/main.go unyil
2021/05/10 19:40:49 the email unyil was not found

Overall program

Previous post
Concurrency Worker Pool Implementation on Golang
Next post
How to Implement Golang's Singleton Design Pattern
comments powered by Disqus