Search on Janik von Rotz

Simple Hugo page search with Lunr.js

Mon, 10 Jun 2019 12:11:31 +0200

Hugo is static website engine and acutally used to generated this blog post. Due to its static nature, there is now way to provide serverside capablities for dynamic lookups such as used by a search feature. Everything dynamically has to be done client side. In this post I am going to setup a search page for a Hugo site the most simple way.

Hugo can not only generate HTML documents, but also JSON documents that can be indexed by Lunr.js. Lunr.js is a small, full-text search library for use in the browser. Using a JSON document and the libararz we are going to create a page for site wide full-text search.

In our scenario we assume that the following Hugo content structure exists:

.
├── content
│   ├── page
│   │   ├── about.md
│   │   ├── archive.md
│   │   └── search.md
│   └── post
│       ├── lithium.md
│       └── markdown.md

There are two sections page and post.

We would like to generate a JSON output for the post section. To do so the supported output format has be enabled in the configuration file.

config.toml

[outputs]
    section = ["JSON", "HTML"]

In addition the current theme has to have a .json list template. Either this template is defined as default template or for the post section only, which I prefer.

themes/THEME_NAME/layouts/post/list.json

[
    {{ range $index, $value := where .Site.Pages "Type" "post" }}
    {{ if $index }}, {{ end }}
    {
        "url": "{{ .RelPermalink }}",
        "title": "{{ .Title }}",
        "content": {{ .Content | plainify | jsonify }}
    }
    {{ end }}
]

This template creates JSON object for each blog post. The url acts as an identifier and the fields title and content will be indexed for the full text search.

If the output configuration has been applied, the template created and Hugo server started, a JSON document should be available at http://localhost:1313/post/index.json. This document will be indexed by Lunr.js.

In our next and final step, we are going to create the acutal search page. In order to understand the logic of the page I have dismantled it into 5 blocks.

content/page/search.md

<script src="https://unpkg.com/lunr/lunr.js"></script>
<script type="text/javascript">
<!--...-->

The Lunr.js library is included at the beginning of the page.

//...
// define globale variables
var idx, searchInput, searchResults = null
var documents = []

function renderSearchResults(results){

    if (results.length > 0) {

        // show max 10 results
        if (results.length > 9){
            results = results.slice(0,10)
        }

        // reset search results
        searchResults.innerHTML = ''

        // append results
        results.forEach(result => {
        
            // create result item
            var article = document.createElement('article')
            article.innerHTML = `
            <a href="${result.ref}"><h3 class="title">${documents[result.ref].title}</h3></a>
            <p><a href="${result.ref}">${result.ref}</a></p>
            <br>
            `
            searchResults.appendChild(article)
        })

    // if results are empty
    } else {
        searchResults.innerHTML = '<p>No results found.</p>'
    }
}
//...

The renderSearchResults processes the search results generated by Lunr.js and appends them on a specific DOM element.

//...
function registerSearchHandler() {

    // register on input event
    searchInput.oninput = function(event) {

        // remove search results if the user empties the search input field
        if (searchInput.value == '') {
            
            searchResults.innerHTML = ''
        } else {
            
            // get input value
            var query = event.target.value

            // run fuzzy search
            var results = idx.search(query + '*')

            // render results
            renderSearchResults(results)
        }
    }

    // set focus on search input and remove loading placeholder
    searchInput.focus()
    searchInput.placeholder = ''
}
//...

The registerSearchHandler function attaches the search call on the input event of the search input field.

//...
window.onload = function() {

    // get dom elements
    searchInput = document.getElementById('search-input')
    searchResults = document.getElementById('search-results')

    // request and index documents
    fetch('/post/index.json', {
        method: 'get'
    }).then(
        res => res.json()
    ).then(
        res => {

            // index document
            idx = lunr(function() {
                this.ref('url')
                this.field('title')
                this.field('content')

                res.forEach(function(doc) {
                    this.add(doc)
                    documents[doc.url] = {
                        'title': doc.title,
                        'content': doc.content,
                    }
                }, this)
            })

            // data is loaded, next register handler
            registerSearchHandler()
        }
    ).catch(
        err => {
            searchResults.innerHTML = `<p>${err}</p>`
        }
    )
}
//...

Once the browser has loaded the DOM structure, the post list is retrieved and indexed.

<!--...-->
</script>

<input id="search-input" type="text" placeholder="Loading..." name="search">

<section id="search-results" class="search"></section>

Only the search input field is actually visible.

This page can be mapped to /search. For a full view of the search.md file, check it out in the example site of the Hugo lithium theme.

Hope this article helped adding a search feature to your Hugo site 😊

And as always remember:

Satisfying solutions call for simple actions.

Sources

Of course I had some help creating this search feature for Hugo. Props to these posts:

Foresty - Build a JSON API With Hugo’s Custom Output Formats
Matt Walters - Hugo and Lunr

Debounce a redux dispatch method in a react component

Thu, 20 Apr 2017 12:46:18 +0000

Tightly connected reactivity in a react application has the side effect that it is sometimes necessary to delay the execution of a method. Assume you have a search input field that filters elements while typing, every field input creates a search request. In order to get rid of unnecessary search requests you have to wait until a user has finished typing and then start the search. To make this work without a search button, you have to intercept repeating executions (debounce) of the search method within a specified time frame and delay the execution of the last call.

Here’s a simple example of a react search component where the redux dispatch method will be debounced.

import React from 'react'
import { Card, CardText, TextField } from 'material-ui'
import { setUserFilter } from '../actions'
import { debounce } from 'lodash'

class UserSearch extends React.Component {

  constructor(props){
    super(props)
    this.updateFilter = debounce(this.updateFilter, 500)
  }

  updateFilter(){
    let { dispatch } = this.props
    let { filter } = this.refs
    dispatch(setUserFilter(filter.getValue()))
  }

  render() {
    return <Card>
      <CardText>

        <TextField
        ref="filter"
        onChange={this.updateFilter.bind(this)} />

        ...

      </CardText>
    </Card>
  }
}

The search is only dispatched if the last call of the update filter method is older than 500ms.

Source: Stack Overflow - Perform debounce in React.js

JavaFX - Filterable and Sortable Tableview with nested Objects

Mon, 30 Nov 2015 10:01:22 +0000

This is a simple example for a sortable and filterable table view with properties from nested objects.

First step design a simple table view like this one:

For this example I use a rather simple approach on the object relationship.

Login 1——1 Person

Every login is added to a person and vice versa. I won’t go into detail, you have make assumptions or adapt other similar cases on this project.

In the person object we have some attributes that we want to display in the table view.

Nachname
Vorname
Email

However the data access object we receive is the login. This is the challenge of nested objects and how to display these properties.

Now we will write the view class for our table view. The view class is connected with the javafx table view.

public class PersonView{

	private static LoginData<Login, Integer> loginData= new LoginData<Login, Integer>(Login.class);
	private FilteredList<Login> filteredData = new FilteredList<Login>(FXCollections.observableArrayList(),	p -> true);

	@FXML
	private TableView<Login> tvData;

	@FXML
	private TextField txFilter;

	@FXML
	private TableColumn<Login, String> tcNachname;

	@FXML
	private TableColumn<Login, String> tcVorname;

	@FXML
	private TableColumn<Login, String> tcEmail;

The LoginData component of course holds the login objects and communicates with the persistence layer. The filtered list will be synced with the table view component. As next, we tell what properties should be displayed in the cells. Remember that we want to display nested properties.

	@Override
	public void initialize(URL arg0, ResourceBundle arg1) {
		
		tcNachname.setCellValueFactory(new Callback<TableColumn.CellDataFeatures<Login,String>,ObservableValue<String>>() {  
			public ObservableValue<String> call(CellDataFeatures<Login, String> param) {
				return new SimpleStringProperty(param.getValue().getPerson().getNachname());
			}  
		});
		
		tcVorname.setCellValueFactory(new Callback<TableColumn.CellDataFeatures<Login,String>,ObservableValue<String>>() {  
			public ObservableValue<String> call(CellDataFeatures<Login, String> param) {
				return new SimpleStringProperty(param.getValue().getPerson().getVorname());
			}  
		});
		tcEmail.setCellValueFactory(new Callback<TableColumn.CellDataFeatures<Login,String>,ObservableValue<String>>() {  
			public ObservableValue<String> call(CellDataFeatures<Login, String> param) {
				return new SimpleStringProperty(param.getValue().getPerson().getEmail());
			}  
		});

By adding a cell value factory you can overwrite the logic of how values of the list should displayed and access. Now it’s time for the filter. By adding a listener to the text property we hook in code that will trigger whenever something is entered into the filter text field.


		txFilter.textProperty().addListener(
				(observable, oldValue, newValue) -> {
					filteredData.setPredicate(t -> {
						
							if (newValue == null || newValue.isEmpty()) {
								return true;
							}

							String lowerCaseFilter = newValue.toLowerCase();
							String objectvalues = t.getPerson().getNachname() 
									+ t.getPerson().getVorname()
									+ t.getPerson().getEmail();
											
							if (objectvalues.toLowerCase().indexOf(lowerCaseFilter) != -1) {
								return true; 
							}

							return false;
						});
				});		

		Refresh();
	}

Finally create a refresh method that loads the data into the table view. Make sure to make this method public.


	public void Refresh() {
		try {	
			List<Login> list = loginData.getAll();
			filteredData = new FilteredList<Login>(FXCollections.observableArrayList(list),	p -> true);
			SortedList<Login> sortedData = new SortedList<Login>(filteredData);
			sortedData.comparatorProperty().bind(tvData.comparatorProperty());
			tvData.setItems(sortedData);
		} catch (Exception e) {
			SomeDialogLauncher.showError(e);
		}
	}
}

Congrats, that’s it.

Let’s write a Laravel application - Fulltext search

Fri, 13 Feb 2015 09:59:27 +0000

This time I want to show you how you add a search functionality to your Laravel application that accomplishes the following features:

Fulltext search in Eloquent model
Define searchable columns
Define result relevance for searchable columns
Customize the look of the search results based on the model
Independent search bar.

To search in Laravel models we will use this searchable trait from Nicolás López. Install it with composer.

First add a new route to handle our search requests.

app/routes.php

Route::get('search', [
    'as' => 'search', 'uses' => 'MemberController@search'
]);

app/models/*.php

For every model that will be searchable you have to add the searchable trait and define the searchable columns and their result relevance. You can get the details of this step from the searchable trait documentation (link above).

You can use any controller of your app to handle the search requests as long you’ll add the following function to this controller.

app/controllers/MemberController.php

	public function search()
	{
		$query = Input::get('q');
		$route = Input::get('route');

		if(preg_match( '/^members.*/', $route)){
			$model = 'Member';
			$route = 'members';
		}elseif(preg_match( '/^categories.*/', $route)){
			$model = 'Category';
			$route = 'categories';
		}elseif(preg_match( '/^boats.*/', $route)){
			$model = 'Boat';
			$route = 'boats';
		}elseif(preg_match( '/^damages.*/', $route)){
			$model = 'Damage';
			$route = 'damages';
		}elseif(preg_match( '/^trips.*/', $route)){
			$model = 'Trip';
			$route = 'trips';
		}elseif(preg_match( '/^users.*/', $route)){
			$model = 'User';
			$route = 'users';
		}else{
			$model = 'Member';
			$route = 'members';
		}

		$results = $model::search($query)->get();
		return View::make('search.index', compact('results', 'model', 'route'));
	}

For every route you want to provide the search functionality you have to add a case and define the model and redirect route.

Now add a new view to display our search results and customize the look of the search results for every model type.

app/views/search/index.blade.php

@extends('default.master')

@section('content')
  @foreach($results as $result)
    <a href="{{ URL::to($route.'/' . $result->id . '/edit') }}">

    @if($model=='Member')
    <article>
      <h2>{{$result->firstname.' '.$result->lastname}}</h2>
      <p>{{$result->id.' '.$result->email}}</p>
    </article>
    @endif

    @if($model=='Category')
    <article>
      <h2>{{$result->name}}</h2>
      <p>{{$result->id}}</p>
    </article>
    @endif

    </a>
  @endforeach
@stop

Finally we add the search bar to a default blade template.

app/views/default/navigation.blade.php

  {{Form::open(['url' => 'search', 'method' => 'GET','class' => 'navbar-form navbar-right', 'role' => 'search'])}}
      <div class="form-group">
        {{ Form::hidden('route', isset($route) ? $route : Route::getCurrentRoute()->getPath())}}
        {{ Form::text('q', null, ['class' => 'form-control', 'placeholder' => 'Search'])}}
      </div>
      {{ Form::button('', ['type' => 'submit', 'class' => 'btn btn-default glyphicon glyphicon-search']) }}
    {{Form::close()}}

If done you’re are ready to run.

Redesign of DuckDuckGo

Tue, 20 May 2014 14:17:15 +0000

The DuckDuckGo search engine website has been redesigned. Hurray!

It grew up from a crappy students project to a respectable Google competitor.

Compared to other search engines DuckDuckGo doesn’t intrude your privacy.

What? You want to know why you should care about your privacy? I’ll let the real pros answer this question: https://duckduckgo.com/privacy

Configure hybrid search results from SharePoint Online in SharePoint on-premise

Wed, 14 May 2014 07:03:29 +0000

This post of is part of my Install SharePoint 2013 Three-tier Farm project. Get the latest version of this article here: https://gist.github.com/10871110

Introduction

In this post I’ll show you how to get search results from your SharePoint Online in your SharePoint 2013 on-premise search center.

Requirements

User synchronisation ActiveDirectory to Office 365 with DirSync
DirSync password sync or ADFS SSO
SharePoint Online
SharePoint 2013 on-premise
- Enterprise Search service
- SharePoint Online Management Shell

Instructions

All configuration will be done either in the Search Administration of the Central Administration or in the PowerShell console of your on-premise SharePoint 2013 server.

Set up Sever to Server Trust

Export certificates

To create a server to server trust we need two certificates.

[certificate name].pfx: In order to replace the STS certificate, the certificate is needed in Personal Information Exchange (PFX) format including the private key.

[certificate name].cer: In order to set up a trust with Office 365 and Windows Azure ACS, the certificate is needed in CER Base64 format.

First launch the Internet Information Services (IIS) Manager
Select your SharePoint web server and double-click Server Certificates
In the Actions pane, click Create Self-Signed Certificate
Enter a name for the certificate and save it with OK
To export the new certificate in the Pfx format select it and click Export in the Actions pane
Fill the fields and click OK Export to: C:\[certificate name].pfx Password: [password]
Also we need to export the certificate in the CER Base64 format. For that purpose make a right-click on the certificate and click on View…
Click the Details tab and then click Copy to File
On the Welcome to the Certificate Export Wizard page, click Next
On the Export Private Key page, click Next
On the Export File Format page, click Base-64 encoded X.509 (.CER), and then click Next.
As file name enter C:\[certificate name].cer and then click Next
Finish the export

Import the new STS (SharePoint Token Service) certificate

Let’s update the certificate on the STS. Configure and run the PowerShell script below on your SharePoint server.

if(-not (Get-PSSnapin "Microsoft.SharePoint.PowerShell" -ErrorAction SilentlyContinue)){Add-PSSnapin "Microsoft.SharePoint.PowerShell"}

# set the cerficates paths and password
$PfxCertPath = "c:\[certificate name].pfx"
$PfxCertPassword = "[password]"
$X64CertPath = "c:\[certificate name].cer"

# get the encrypted pfx certificate object
$PfxCert = New-Object System.Security.Cryptography.X509Certificates.X509Certificate2 $PfxCertPath, $PfxCertPassword, 20

# import it
Set-SPSecurityTokenServiceConfig -ImportSigningCertificate $PfxCert

Type Yes when prompted with the following message.

You are about to change the signing certificate for the Security Token Service. Changing the certificate to an invalid, inaccessible or non-existent certificate will cause your SharePoint installation to stop functioning. Refer to the following article for instructions on how to change this certificate: http://go.microsoft.com/fwlink/?LinkID=178475. Are you sure, you want to continue?

Restart IIS so STS picks up the new certificate.

& iisreset
& net stop SPTimerV4
& net start SPTimerV4

Now validate the certificate replacement by running several PowerShell commands and compare their outputs.

# set the cerficates paths and password
$PfxCertPath = "c:\[certificate name].pfx"
$PfxCertPassword = "[password]"

# get the encrypted pfx certificate object
New-Object System.Security.Cryptography.X509Certificates.X509Certificate2 $PfxCertPath, $PfxCertPassword, 20

# compare the output above with this output
(Get-SPSecurityTokenServiceConfig).LocalLoginProvider.SigningCertificate

Establish the server to server trust

if(-not (Get-PSSnapin "Microsoft.SharePoint.PowerShell" -ErrorAction SilentlyContinue)){Add-PSSnapin "Microsoft.SharePoint.PowerShell"}
Import-Module MSOnline 
Import-Module MSOnlineExtended

# set the cerficates paths and password
$PfxCertPath = "c:\[certificate name].pfx"
$PfxCertPassword = "[password]"
$X64CertPath = "c:\[certificate name].cer"

# set the onpremise domain that you added to Office 365
$SPCN = "sharepoint.domain.com" 

# your onpremise SharePoint site url
$SPSite="http://sharepoint"

# don't change this value
$SPOAppID="00000003-0000-0ff1-ce00-000000000000"

# get the encrypted pfx certificate object
$PfxCert = New-Object System.Security.Cryptography.X509Certificates.X509Certificate2 $PfxCertPath, $PfxCertPassword, 20

# get the raw data
$PfxCertBin = $PfxCert.GetRawCertData()

# create a new certificate object
$X64Cert = New-Object System.Security.Cryptography.X509Certificates.X509Certificate2

# import the base 64 encoded certificate
$X64Cert.Import($X64CertPath)

# get the raw data
$X64CertBin = $X64Cert.GetRawCertData()

# save base 64 string in variable
$CredValue = [System.Convert]::ToBase64String($X64CertBin)

# connect to office 3656
Connect-MsolService

# register the on-premise STS as service principal in Office 365

# add a new service principal
New-MsolServicePrincipalCredential -AppPrincipalId $SPOAppID -Type asymmetric -Usage Verify -Value $CredValue
$MsolServicePrincipal = Get-MsolServicePrincipal -AppPrincipalId $SPOAppID
$SPServicePrincipalNames = $MsolServicePrincipal.ServicePrincipalNames
$SPServicePrincipalNames.Add("$SPOAppID/$SPCN")
Set-MsolServicePrincipal -AppPrincipalId $SPOAppID -ServicePrincipalNames $SPServicePrincipalNames

# get the online name identifier
$MsolCompanyInformationID = (Get-MsolCompanyInformation).ObjectID
$MsolServicePrincipalID = (Get-MsolServicePrincipal -ServicePrincipalName $SPOAppID).ObjectID
$MsolNameIdentifier = "$MsolServicePrincipalID@$MsolCompanyInformationID"

# establish the trust from on-premise with ACS (Azure Control Service)

# add a new authenticatio realm
$SPSite = Get-SPSite $SPSite
$SPAppPrincipal = Register-SPAppPrincipal -site $SPSite.rootweb -nameIdentifier $MsolNameIdentifier -displayName "SharePoint Online"
Set-SPAuthenticationRealm -realm $MsolServicePrincipalID

# register the ACS application proxy and token issuer
New-SPAzureAccessControlServiceApplicationProxy -Name "ACS" -MetadataServiceEndpointUri "https://accounts.accesscontrol.windows.net/metadata/json/1/" -DefaultProxyGroup
New-SPTrustedSecurityTokenIssuer -MetadataEndpoint "https://accounts.accesscontrol.windows.net/metadata/json/1/" -IsTrustBroker -Name "ACS"

Add a new result source

To get search results from SharePoint Online we have to add a new result source. Run the following script in a PowerShell ISE session on your SharePoint 2013 on-premise server. Don’t forget to update the settings region

if(-not (Get-PSSnapin "Microsoft.SharePoint.PowerShell" -ErrorAction SilentlyContinue)){Add-PSSnapin "Microsoft.SharePoint.PowerShell"}

# region settings 
$RemoteSharePointUrl = "http://[example].sharepoint.com"
$ResultSourceName = "SharePoint Online"
$QueryTransform = "{searchTerms}"
$Provier = "SharePoint-Remoteanbieter"
# region settings end

$SPEnterpriseSearchServiceApplication = Get-SPEnterpriseSearchServiceApplication
$FederationManager = New-Object Microsoft.Office.Server.Search.Administration.Query.FederationManager($SPEnterpriseSearchServiceApplication)
$SPEnterpriseSearchOwner = Get-SPEnterpriseSearchOwner -Level Ssa  

$ResultSource = $FederationManager.GetSourceByName($ResultSourceName, $SPEnterpriseSearchOwner)
if(!$ResultSource){
    Write-Host "Result source does not exist. Creating..."
    $ResultSource = $FederationManager.CreateSource($SPEnterpriseSearchOwner)
}

$ResultSource.Name = $ResultSourceName
$ResultSource.ProviderId = $FederationManager.ListProviders()[$Provier].Id
$ResultSource.ConnectionUrlTemplate = $RemoteSharePointUrl
$ResultSource.CreateQueryTransform($QueryTransform)
$ResultSource.Commit()

Add a new query rule

In the Search Administration click on Query Rules
Select Local SharePoint as Result Source
Click New Query Rule
Enter a Rule name f.g. Search results from SharePoint Online
Expand the Context section
Under Query is performed on these sources click on Add Source
Select your SharePoint Online result source
In the Query Conditions section click on Remove Condition
In the Actions section click on Add Result Block
As title enter Results for “{subjectTerms}” from SharePoint Online
In the Search this Source dropdown select your SharePoint Online result source
Select 3 in the Items dropdown
Expand the Settings section and select “More” link goes to the following URL
In the box below enter this Url https://[example].sharepoint.com/search/pages/results.aspx?k={subjectTerms}
Select This block is always shown above core results and click the OK button
Save the new query rule

Source

Display hybrid search results in SharePoint Server 2013
Office 365-Configure Hybrid Search with Directory Synchronization –Password Sync
Office 365-Configure Hybrid Search with Directory Synchronization –Password Sync –Part2

Add View Folder link to search results page in SharePoint

Mon, 23 Sep 2013 11:19:09 +0000

In the default document search results in the SharePoint search center it is only possible to open the documents. It is not possible to navigate to the documents location.

in the results.aspx file (accessible with the SharePoint designer) make a replacement as showed:

https://gist.github.com/janikvonrotz/6668613